Перейти к содержанию

Оптимизация пропускной способности Qwen3-32B на NVIDIA H100 GPU

Вывод

qwen3-32b-h100

Рекомендуемая конфигурация для оптимизации пропускной способности Qwen3-32B на H100 GPU:

Команда обслуживания модели 
1
trtllm-serve Qwen/Qwen3-32B-FP8 --max_seq_len=40960 --enable_chunked_prefill

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка базовая конфигурация (vLLM без оптимизации) После оптимизации
ShareGPT Общий TPS: 2352.82
Среднее TPOT(ms): 88.09		 Общий TPS: 4284.68 (+82.1%)
Среднее TPOT(ms): 158.64
Короткий запрос Общий TPS: 4005.62
Среднее TPOT(ms): 1169.72		 Общий TPS: 6826.33 (+70.4%)
Среднее TPOT(ms): 890.11
Запрос средней длины Общий TPS: 3195.30
Среднее TPOT(ms): 86.62		 Общий TPS: 5901.95 (+84.7%)
Среднее TPOT(ms): 223.40
Длинный запрос Общий TPS: 2148.69
Среднее TPOT(ms): 66.61		 Общий TPS: 4929.16 (+129.4%)
Среднее TPOT(ms): 138.75
Очень длинный запрос Общий TPS: 2771.81
Среднее TPOT(ms): 43.42		 Общий TPS: 4068.16 (+10.7%)
Среднее TPOT(ms): 286.97

Примечание

  1. Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
  2. Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph, Torch Compile и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
  3. Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.
  4. Квантизация может влиять на точность, однако для большинства моделей FP8 обеспечивает снижение точности менее чем на 1%. Подробности см. в результатах оценки. Поэтому для обслуживания с высокой пропускной способностью настоятельно рекомендуется квантизация FP8.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Цель оптимизации

Высокая пропускная способность при высокой конкурентности запросов.

Условия эксперимента

Модель

Qwen3-32B

Оборудование

Один NVIDIA H100 SXM GPU.

Версия движка

  • vLLM: v0.11.0
  • SGLang: v0.5.5.post1
  • TensorRT-LLM: v1.2.0rc1

Набор данных бенчмарка

  1. ShareGPT
  2. Случайный набор данных с различной длиной последовательностей:
    • Очень длинный запрос: 32000 входных токенов, 100 выходных токенов
    • Длинный запрос: 4000 входных токенов, 200 выходных токенов
    • Запрос средней длины: 2000 входных токенов, 100 выходных токенов
    • Короткий запрос: 128 входных токенов, 4 выходных токенов

Скрипт бенчмарка

Для оценки производительности модели используется vLLM bench CLI. Бенчмарк запускается следующей командой:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-32B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-32B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Результаты эксперимента

1. Выбор движка инференса

vLLM

Скрипт обслуживания модели 
1
vllm serve Qwen/Qwen3-32B
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  178.01
Total input tokens:                      217393
Total generated tokens:                  201422
Request throughput (req/s):              5.62
Output token throughput (tok/s):         1131.55
Peak output token throughput (tok/s):    2360.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2352.82
---------------Time to First Token----------------
Mean TTFT (ms):                          78118.74
Median TTFT (ms):                        77291.46
P99 TTFT (ms):                           153161.51
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          88.09
Median TPOT (ms):                        70.72
P99 TPOT (ms):                           378.18
---------------Inter-token Latency----------------
Mean ITL (ms):                           69.76
Median ITL (ms):                         50.20
P99 ITL (ms):                            277.93
==================================================

SGLang

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-32B
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  247.94
Total input tokens:                      217393
Total generated tokens:                  201826
Request throughput (req/s):              4.03
Output token throughput (tok/s):         814.03
Peak output token throughput (tok/s):    1329.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          1690.84
---------------Time to First Token----------------
Mean TTFT (ms):                          115690.30
Median TTFT (ms):                        117738.49
P99 TTFT (ms):                           230503.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          57.86
Median TPOT (ms):                        54.19
P99 TPOT (ms):                           95.84
---------------Inter-token Latency----------------
Mean ITL (ms):                           56.25
Median ITL (ms):                         43.94
P99 ITL (ms):                            207.79
==================================================

TensorRT-LLM

Скрипт обслуживания модели 
1
trtllm-serve Qwen/Qwen3-32B
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  157.59
Total input tokens:                      217393
Total generated tokens:                  201723
Request throughput (req/s):              6.35
Output token throughput (tok/s):         1280.04
Peak output token throughput (tok/s):    1958.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2659.50
---------------Time to First Token----------------
Mean TTFT (ms):                          65281.52
Median TTFT (ms):                        65325.76
P99 TTFT (ms):                           133106.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          61.90
Median TPOT (ms):                        59.59
P99 TPOT (ms):                           121.97
---------------Inter-token Latency----------------
Mean ITL (ms):                           59.22
Median ITL (ms):                         47.89
P99 ITL (ms):                            214.01
==================================================

Результат: TensorRT-LLM (2659.50 tok/s) > vLLM (2352.82 tok/s) > SGLang (1690.84 tok/s)

2. Quantization в TensorRT-LLM

FP8

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-32B-FP8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  104.55
Total input tokens:                      217393
Total generated tokens:                  201376
Request throughput (req/s):              9.56
Output token throughput (tok/s):         1926.06
Peak output token throughput (tok/s):    4391.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4005.32
---------------Time to First Token----------------
Mean TTFT (ms):                          35510.55
Median TTFT (ms):                        31577.26
P99 TTFT (ms):                           79653.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          205.13
Median TPOT (ms):                        119.96
P99 TPOT (ms):                           1141.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           118.32
Median ITL (ms):                         86.53
P99 ITL (ms):                            1195.28
==================================================

3. Включение блочного prefill в TensorRT-LLM

Скрипт обслуживания модели 
1
trtllm-serve Qwen/Qwen3-32B-FP8 --max_seq_len=40960 --enable_chunked_prefill
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  97.74
Total input tokens:                      217393
Total generated tokens:                  201376
Request throughput (req/s):              10.23
Output token throughput (tok/s):         2060.40
Peak output token throughput (tok/s):    4371.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4284.68
---------------Time to First Token----------------
Mean TTFT (ms):                          31208.75
Median TTFT (ms):                        25003.73
P99 TTFT (ms):                           72827.99
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          158.64
Median TPOT (ms):                        115.67
P99 TPOT (ms):                           993.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           107.04
Median ITL (ms):                         85.28
P99 ITL (ms):                            569.95
==================================================

Сводка методов оптимизации

Метод оптимизации Прирост пропускной способности
Выбор движка +13.0%
Quantization +50.6%
Включение блочного prefill +7.0%

Другие сценарии бенчмарка

Мы дополнительно проверили оптимизированную конфигурацию на разных нагрузках, чтобы оценить её обобщаемость.

Базовый скрипт обслуживания модели 
1
vllm serve Qwen/Qwen3-32B
Результаты базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  1157.25
Total input tokens:                      3200000
Total generated tokens:                  7681
Request throughput (req/s):              0.09
Output token throughput (tok/s):         6.64
Peak output token throughput (tok/s):    24.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          2771.81
---------------Time to First Token----------------
Mean TTFT (ms):                          585981.22
Median TTFT (ms):                        582757.65
P99 TTFT (ms):                           1144686.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          43.42
Median TPOT (ms):                        43.36
P99 TPOT (ms):                           44.34
---------------Inter-token Latency----------------
Mean ITL (ms):                           42.86
Median ITL (ms):                         43.30
P99 ITL (ms):                            46.54
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  976.29
Total input tokens:                      1997942
Total generated tokens:                  99790
Request throughput (req/s):              0.51
Output token throughput (tok/s):         102.21
Peak output token throughput (tok/s):    186.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          2148.69
---------------Time to First Token----------------
Mean TTFT (ms):                          482284.21
Median TTFT (ms):                        483375.28
P99 TTFT (ms):                           956141.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          66.61
Median TPOT (ms):                        62.08
P99 TPOT (ms):                           100.86
---------------Inter-token Latency----------------
Mean ITL (ms):                           66.29
Median ITL (ms):                         43.38
P99 ITL (ms):                            1395.23
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  328.04
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              1.52
Output token throughput (tok/s):         152.42
Peak output token throughput (tok/s):    359.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          3195.30
---------------Time to First Token----------------
Mean TTFT (ms):                          163909.40
Median TTFT (ms):                        164148.38
P99 TTFT (ms):                           322980.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          86.62
Median TPOT (ms):                        84.24
P99 TPOT (ms):                           131.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           85.97
Median ITL (ms):                         44.80
P99 ITL (ms):                            1462.05
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  32.88
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              30.42
Output token throughput (tok/s):         121.66
Peak output token throughput (tok/s):    522.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4005.62
---------------Time to First Token----------------
Mean TTFT (ms):                          21648.67
Median TTFT (ms):                        21342.36
P99 TTFT (ms):                           32678.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1169.72
Median TPOT (ms):                        1292.78
P99 TPOT (ms):                           1417.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           877.29
Median ITL (ms):                         1141.11
P99 ITL (ms):                            1435.80
==================================================
Скрипт обслуживания после оптимизации 
1
trtllm-serve Qwen/Qwen3-32B-FP8 --max_seq_len=40960 --enable_chunked_prefill
Результаты бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  788.52
Total input tokens:                      3200000
Total generated tokens:                  7832
Request throughput (req/s):              0.13
Output token throughput (tok/s):         9.93
Peak output token throughput (tok/s):    104.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          4068.16
---------------Time to First Token----------------
Mean TTFT (ms):                          396294.90
Median TTFT (ms):                        397717.15
P99 TTFT (ms):                           778540.06
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          286.97
Median TPOT (ms):                        255.80
P99 TPOT (ms):                           581.09
---------------Inter-token Latency----------------
Mean ITL (ms):                           284.98
Median ITL (ms):                         39.76
P99 ITL (ms):                            2105.62
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  425.60
Total input tokens:                      1997942
Total generated tokens:                  99888
Request throughput (req/s):              1.17
Output token throughput (tok/s):         234.70
Peak output token throughput (tok/s):    828.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          4929.16
---------------Time to First Token----------------
Mean TTFT (ms):                          208082.20
Median TTFT (ms):                        204136.20
P99 TTFT (ms):                           415000.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          138.75
Median TPOT (ms):                        141.52
P99 TPOT (ms):                           151.74
---------------Inter-token Latency----------------
Mean ITL (ms):                           138.01
Median ITL (ms):                         45.37
P99 ITL (ms):                            1254.20
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  177.60
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              2.82
Output token throughput (tok/s):         281.53
Peak output token throughput (tok/s):    1529.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          5901.95
---------------Time to First Token----------------
Mean TTFT (ms):                          86596.60
Median TTFT (ms):                        86629.72
P99 TTFT (ms):                           171842.84
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          223.40
Median TPOT (ms):                        237.63
P99 TPOT (ms):                           249.66
---------------Inter-token Latency----------------
Mean ITL (ms):                           221.16
Median ITL (ms):                         49.40
P99 ITL (ms):                            1216.32
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  19.29
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              51.83
Output token throughput (tok/s):         207.33
Peak output token throughput (tok/s):    860.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6826.33
---------------Time to First Token----------------
Mean TTFT (ms):                          10718.84
Median TTFT (ms):                        10511.60
P99 TTFT (ms):                           19055.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          890.11
Median TPOT (ms):                        949.23
P99 TPOT (ms):                           1351.38
---------------Inter-token Latency----------------
Mean ITL (ms):                           667.57
Median ITL (ms):                         925.89
P99 ITL (ms):                            1854.09
==================================================