Online Installation
Supported Devices
- AMD GPUs
Supported Platforms
OS | Version | Arch | Supported methods |
---|---|---|---|
Linux | Ubuntu 22.04 Ubuntu 24.04 |
AMD64 | Docker Installation (Recommended) Installation Script |
Windows | 10 11 Server 2022 |
AMD64 | Installation Script |
Prerequisites
- Port Requirements
- CPU support for llama-box backend: AMD64 with AVX2
Check if the CPU is supported:
lscpu | grep avx2
Windows users need to manually verify support for the above instructions.
Docker Installation
Supported Devices
Devices | Supported Backends |
---|---|
gfx1100: AMD Radeon RX 7900 XTX/7900 XT/7900 GRE | llama-box, vLLM |
gfx1101: AMD Radeon RX 7800 XT/7700 XT | llama-box, vLLM |
gfx1102: AMD Radeon RX 7600 XT/7600 | llama-box, vLLM |
gfx942: AMD Instinct MI300X/MI300A | llama-box, vLLM |
gfx90a: AMD Instinct MI250X/MI250/MI210 | llama-box, vLLM |
gfx1030: AMD Radeon RX 6950 XT/6900 XT/6800 XT/6800 | llama-box |
gfx1031: AMD Radeon RX 6750 XT/6700 XT/6700 | llama-box |
gfx1032: AMD Radeon RX 6650 XT/6600 XT/6600 | llama-box |
gfx908: AMD Instinct MI100 | llama-box |
gfx906: AMD Instinct MI50 | llama-box |
Prerequisites
Select the appropriate installation method for your system. Here, we provide steps for Linux (Ubuntu 22.04). For other systems, refer to the ROCm documentation.
- Install ROCm:
sudo apt update
wget https://repo.radeon.com/amdgpu-install/6.2.4/ubuntu/jammy/amdgpu-install_6.2.60204-1_all.deb
sudo apt install ./amdgpu-install_6.2.60204-1_all.deb
amdgpu-install -y --usecase=graphics,rocm
sudo reboot
- Set Groups permissions:
sudo usermod -a -G render,video $LOGNAME
sudo reboot
- Verify Installation:
Verify that the current user is added to the render and video groups:
groups
Check if amdgpu kernel driver is installed:
dkms status
Check if the GPU is listed as an agent:
rocminfo
Check rocm-smi
:
rocm-smi -i --showmeminfo vram --showpower --showserial --showuse --showtemp --showproductname --showuniqueid --json
Run GPUStack
Run the following command to start the GPUStack server and built-in worker (host network mode is recommended):
docker run -d --name gpustack \
--restart=unless-stopped \
--device=/dev/kfd \
--device=/dev/dri \
--network=host \
--ipc=host \
--group-add video \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v gpustack-data:/var/lib/gpustack \
gpustack/gpustack:latest-rocm
If you need to change the default server port 80, please use the --port
parameter:
docker run -d --name gpustack \
--restart=unless-stopped \
--device=/dev/kfd \
--device=/dev/dri \
--network=host \
--ipc=host \
--group-add video \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v gpustack-data:/var/lib/gpustack \
gpustack/gpustack:latest-rocm \
--port 9090
If other ports are in conflict, or if you want to customize startup options, refer to the CLI Reference for available flags and configuration instructions.
Check if the startup logs are normal:
docker logs -f gpustack
If the logs are normal, open http://your_host_ip
in the browser to access the GPUStack UI. Log in to GPUStack with username admin
and the default password. You can run the following command to get the password for the default setup:
docker exec -it gpustack cat /var/lib/gpustack/initial_admin_password
(Optional) Add Worker
You can add more GPU nodes to GPUStack to form a GPU cluster. You need to add workers on other GPU nodes and specify the --server-url
and --token
parameters to join GPUStack.
To get the token used for adding workers, run the following command on the GPUStack server node:
docker exec -it gpustack cat /var/lib/gpustack/token
To start GPUStack as a worker, and register it with the GPUStack server, run the following command on the worker node. Be sure to replace the URL and token with your specific values:
docker run -d --name gpustack \
--restart=unless-stopped \
--device=/dev/kfd \
--device=/dev/dri \
--network=host \
--ipc=host \
--group-add video \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v gpustack-data:/var/lib/gpustack \
gpustack/gpustack:latest-rocm \
--server-url http://your_gpustack_url --token your_gpustack_token
Note
-
Heterogeneous cluster is supported. No matter what type of device it is, you can add it to the current GPUStack as a worker by specifying the
--server-url
and--token
parameters. -
You can set additional flags for the
gpustack start
command by appending them to the docker run command. For configuration details, please refer to the CLI Reference. -
You can either use the
--ipc=host
flag or--shm-size
flag to allow the container to access the host’s shared memory. It is used by vLLM and pyTorch to share data between processes under the hood, particularly for tensor parallel inference.
Installation Script
Supported Devices
Devices | Supported Backends |
---|---|
gfx1100: AMD Radeon RX 7900 XTX/7900 XT/7900 GRE | llama-box |
gfx1101: AMD Radeon RX 7800 XT/7700 XT | llama-box |
gfx1102: AMD Radeon RX 7600 XT/7600 | llama-box |
gfx1030: AMD Radeon RX 6950 XT/6900 XT/6800 XT/6800 | llama-box |
gfx1031: AMD Radeon RX 6750 XT/6700 XT/6700 | llama-box |
gfx1032: AMD Radeon RX 6650 XT/6600 XT/6600 | llama-box |
gfx942: AMD Instinct MI300X/MI300A | llama-box |
gfx90a: AMD Instinct MI250X/MI250/MI210 | llama-box |
gfx908: AMD Instinct MI100 | llama-box |
gfx906: AMD Instinct MI50 | llama-box |
View more details here.
Devices | Supported Backends |
---|---|
gfx1100: AMD Radeon RX 7900 XTX/7900 XT | llama-box |
gfx1101: AMD Radeon RX 7800 XT/7700 XT | llama-box |
gfx1102: AMD Radeon RX 7600 XT/7600 | llama-box |
gfx1030: AMD Radeon RX 6950 XT/6900 XT/6800 XT/6800 | llama-box |
gfx1031: AMD Radeon RX 6750 XT/6700 XT/6700 | llama-box |
gfx1032: AMD Radeon RX 6650 XT/6600 XT/6600 | llama-box |
View more details here.
Prerequisites
- Install ROCm 6.2.4:
sudo apt update
wget https://repo.radeon.com/amdgpu-install/6.2.4/ubuntu/jammy/amdgpu-install_6.2.60204-1_all.deb
sudo apt install ./amdgpu-install_6.2.60204-1_all.deb
amdgpu-install -y --usecase=graphics,rocm
sudo reboot
- Set Groups permissions:
sudo usermod -a -G render,video $LOGNAME
sudo reboot
- Verify Installation:
Verify that the current user is added to the render and video groups:
groups
Check if amdgpu kernel driver is installed:
dkms status
Check if the GPU is listed as an agent:
rocminfo
Check rocm-smi
:
rocm-smi -i --showmeminfo vram --showpower --showserial --showuse --showtemp --showproductname --showuniqueid --json
- Type the following command on your system from PowerShell to confirm that the obtained information matches that listed in Supported SKUs:
Get-ComputerInfo | Format-Table CsSystemType,OSName,OSDisplayVersion
-
Download the installer from the HIP-SDK download page.
-
Launch the installer.
Run GPUStack
GPUStack provides a script to install it as a service with default port 80.
curl -sfL https://get.gpustack.ai | sh -s -
To configure additional environment variables and startup flags when running the script, refer to the Installation Script.
After installed, ensure that the GPUStack startup logs are normal:
tail -200f /var/log/gpustack.log
If the startup logs are normal, open http://your_host_ip
in the browser to access the GPUStack UI. Log in to GPUStack with username admin
and the default password. You can run the following command to get the password for the default setup:
cat /var/lib/gpustack/initial_admin_password
If you specify the --data-dir
parameter to set the data directory, the initial_admin_password
file will be located in the specified directory.
Invoke-Expression (Invoke-WebRequest -Uri "https://get.gpustack.ai" -UseBasicParsing).Content
To configure additional environment variables and startup flags when running the script, refer to the Installation Script.
After installed, ensure that the GPUStack startup logs are normal:
Get-Content "$env:APPDATA\gpustack\log\gpustack.log" -Tail 200 -Wait
If the startup logs are normal, open http://your_host_ip
in the browser to access the GPUStack UI. Log in to GPUStack with username admin
and the default password. You can run the following command to get the password for the default setup:
Get-Content -Path "$env:APPDATA\gpustack\initial_admin_password" -Raw
--data-dir
parameter to set the data directory, the initial_admin_password
file will be located in the specified directory.
(Optional) Add Worker
To add workers to the GPUStack cluster, you need to specify the server URL and authentication token when installing GPUStack on the workers.
To get the token used for adding workers, run the following command on the GPUStack server node:
cat /var/lib/gpustack/token
If you specify the --data-dir
parameter to set the data directory, the token
file will be located in the specified directory.
To install GPUStack and start it as a worker, and register it with the GPUStack server, run the following command on the worker node. Be sure to replace the URL and token with your specific values:
curl -sfL https://get.gpustack.ai | sh -s - --server-url http://your_gpustack_url --token your_gpustack_token
After installed, ensure that the GPUStack startup logs are normal:
tail -200f /var/log/gpustack.log
To add workers to the GPUStack cluster, you need to specify the server URL and authentication token when installing GPUStack on the workers.
To get the token used for adding workers, run the following command on the GPUStack server node:
Get-Content -Path "$env:APPDATA\gpustack\token" -Raw
If you specify the --data-dir
parameter to set the data directory, the token
file will be located in the specified directory.
To install GPUStack and start it as a worker, and register it with the GPUStack server, run the following command on the worker node. Be sure to replace the URL and token with your specific values:
Invoke-Expression "& { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --server-url http://your_gpustack_url --token your_gpustack_token"
After installed, ensure that the GPUStack startup logs are normal:
Get-Content "$env:APPDATA\gpustack\log\gpustack.log" -Tail 200 -Wait