Перейти к содержанию

Оптимизация пропускной способности Qwen3-14B на NVIDIA A100 GPU

Вывод

qwen3-14b-a100

Рекомендуемая конфигурация для оптимизации пропускной способности Qwen3-14B на A100 GPU:

Команда обслуживания модели 
1
trtllm-serve Qwen/Qwen3-14B --max_seq_len=40960 --enable_chunked_prefill

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка базовая конфигурация (vLLM без оптимизации) После оптимизации
ShareGPT Общий TPS: 3922.41
Среднее TPOT(ms): 114.34		 Общий TPS: 5770.24 (+47.1%)
Среднее TPOT(ms): 260.46
Короткий запрос Общий TPS: 4916.63
Среднее TPOT(ms): 233.35		 Общий TPS: 8762.17 (+78.2%)
Среднее TPOT(ms): 715.19
Запрос средней длины Общий TPS: 5967.13
Среднее TPOT(ms): 301.73		 Общий TPS: 7474.00 (+25.3%)
Среднее TPOT(ms): 313.69
Длинный запрос Общий TPS: 5153.42
Среднее TPOT(ms): 254.61		 Общий TPS: 6418.65 (+24.6%)
Среднее TPOT(ms): 196.10
Очень длинный запрос Общий TPS: 4430.14
Среднее TPOT(ms): 445.58		 Общий TPS: 4413.82 (-0.4%)
Среднее TPOT(ms): 493.18

Примечание

  1. Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
  2. Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph, Torch Compile и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
  3. Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Цель оптимизации

Высокая пропускная способность при высокой конкурентности запросов.

Условия эксперимента

Модель

Qwen3-14B

Оборудование

Один NVIDIA A100 SXM GPU.

Версия движка

  • vLLM: v0.11.0
  • SGLang: v0.5.5.post1
  • TensorRT-LLM: v1.2.0rc1

Набор данных бенчмарка

  1. ShareGPT
  2. Случайный набор данных с различной длиной последовательностей:
    • Очень длинный запрос: 32000 входных токенов, 100 выходных токенов
    • Длинный запрос: 4000 входных токенов, 200 выходных токенов
    • Запрос средней длины: 2000 входных токенов, 100 выходных токенов
    • Короткий запрос: 128 входных токенов, 4 выходных токенов

Скрипт бенчмарка

Для оценки производительности модели используется vLLM bench CLI. Бенчмарк запускается следующей командой:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-14B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-14B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Результаты эксперимента

1. Выбор движка инференса

vLLM

Скрипт обслуживания модели 
1
vllm serve Qwen/Qwen3-14B
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  106.82
Total input tokens:                      217393
Total generated tokens:                  201605
Request throughput (req/s):              9.36
Output token throughput (tok/s):         1887.31
Peak output token throughput (tok/s):    4616.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          3922.41
---------------Time to First Token----------------
Mean TTFT (ms):                          41875.09
Median TTFT (ms):                        39151.17
P99 TTFT (ms):                           83576.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          114.34
Median TPOT (ms):                        106.15
P99 TPOT (ms):                           262.52
---------------Inter-token Latency----------------
Mean ITL (ms):                           95.04
Median ITL (ms):                         85.55
P99 ITL (ms):                            271.69
==================================================

SGLang

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-14B
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  79.35
Total input tokens:                      217393
Total generated tokens:                  201706
Request throughput (req/s):              12.60
Output token throughput (tok/s):         2542.06
Peak output token throughput (tok/s):    6721.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5281.81
---------------Time to First Token----------------
Mean TTFT (ms):                          20365.34
Median TTFT (ms):                        19226.41
P99 TTFT (ms):                           37314.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          364.93
Median TPOT (ms):                        185.18
P99 TPOT (ms):                           2181.56
---------------Inter-token Latency----------------
Mean ITL (ms):                           141.30
Median ITL (ms):                         84.71
P99 ITL (ms):                            566.50
==================================================

TensorRT-LLM

Скрипт обслуживания модели 
1
trtllm-serve Qwen/Qwen3-14B
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  72.71
Total input tokens:                      217393
Total generated tokens:                  201471
Request throughput (req/s):              13.75
Output token throughput (tok/s):         2770.99
Peak output token throughput (tok/s):    7336.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5760.97
---------------Time to First Token----------------
Mean TTFT (ms):                          15258.26
Median TTFT (ms):                        14210.29
P99 TTFT (ms):                           37520.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          261.77
Median TPOT (ms):                        166.55
P99 TPOT (ms):                           915.21
---------------Inter-token Latency----------------
Mean ITL (ms):                           130.55
Median ITL (ms):                         85.40
P99 ITL (ms):                            931.59
==================================================

Результат: TensorRT-LLM (5760.97 tok/s) > SGLang (5281.81 tok/s) > vLLM (3922.41 tok/s)

2. Quantization в TensorRT-LLM/vLLM

TensorRT-LLM FP8

Скрипт обслуживания модели 
1
trtllm-serve Qwen/Qwen3-14B-FP8
Результат бенчмарка 
1
# RuntimeError: Unsupported SM version for FP8 block scaling GEMM

TensorRT-LLM AWQ

Скрипт обслуживания модели 
1
trtllm-serve Qwen/Qwen3-14B-AWQ
Результат бенчмарка 
1
# KeyError: 'weight'

vLLM FP8

Скрипт обслуживания модели 
1
vllm serve Qwen/Qwen3-14B-FP8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  125.66
Total input tokens:                      217393
Total generated tokens:                  201655
Request throughput (req/s):              7.96
Output token throughput (tok/s):         1604.75
Peak output token throughput (tok/s):    3823.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          3334.74
---------------Time to First Token----------------
Mean TTFT (ms):                          52287.47
Median TTFT (ms):                        48838.08
P99 TTFT (ms):                           104986.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          146.36
Median TPOT (ms):                        132.69
P99 TPOT (ms):                           372.46
---------------Inter-token Latency----------------
Mean ITL (ms):                           118.59
Median ITL (ms):                         96.50
P99 ITL (ms):                            380.19
==================================================

vLLM AWQ

Скрипт обслуживания модели 
1
vllm serve Qwen/Qwen3-14B-AWQ
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  111.81
Total input tokens:                      217393
Total generated tokens:                  201461
Request throughput (req/s):              8.94
Output token throughput (tok/s):         1801.74
Peak output token throughput (tok/s):    4069.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          3745.97
---------------Time to First Token----------------
Mean TTFT (ms):                          47136.93
Median TTFT (ms):                        44343.62
P99 TTFT (ms):                           94568.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          129.77
Median TPOT (ms):                        120.87
P99 TPOT (ms):                           331.28
---------------Inter-token Latency----------------
Mean ITL (ms):                           106.86
Median ITL (ms):                         89.48
P99 ITL (ms):                            338.24
==================================================

3. Максимальное число токенов в батче в TensorRT-LLM

Скрипт обслуживания модели 
1
trtllm-serve Qwen/Qwen3-14B --max_num_tokens=16384
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# --max_num_tokens=16384
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  72.54
Total input tokens:                      217393
Total generated tokens:                  201631
Request throughput (req/s):              13.79
Output token throughput (tok/s):         2779.60
Peak output token throughput (tok/s):    5583.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5776.50
---------------Time to First Token----------------
Mean TTFT (ms):                          15374.97
Median TTFT (ms):                        13585.74
P99 TTFT (ms):                           38913.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          268.46
Median TPOT (ms):                        159.86
P99 TPOT (ms):                           1618.81
---------------Inter-token Latency----------------
Mean ITL (ms):                           129.13
Median ITL (ms):                         87.94
P99 ITL (ms):                            1712.52
==================================================

# --max_num_tokens=32768
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  74.21
Total input tokens:                      217393
Total generated tokens:                  201567
Request throughput (req/s):              13.48
Output token throughput (tok/s):         2716.13
Peak output token throughput (tok/s):    5559.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5645.51
---------------Time to First Token----------------
Mean TTFT (ms):                          16776.11
Median TTFT (ms):                        15050.03
P99 TTFT (ms):                           43000.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          234.79
Median TPOT (ms):                        150.33
P99 TPOT (ms):                           1486.15
---------------Inter-token Latency----------------
Mean ITL (ms):                           123.01
Median ITL (ms):                         87.61
P99 ITL (ms):                            525.09
==================================================

Сводка методов оптимизации

Метод оптимизации Прирост пропускной способности
Выбор движка +47.1%
Quantization -
Максимальное число токенов в батче -

Другие сценарии бенчмарка

Мы дополнительно проверили оптимизированную конфигурацию на разных нагрузках, чтобы оценить её обобщаемость.

Базовый скрипт обслуживания модели 
1
vllm serve Qwen/Qwen3-14B
Результаты базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  724.56
Total input tokens:                      3200000
Total generated tokens:                  9891
Request throughput (req/s):              0.14
Output token throughput (tok/s):         13.65
Peak output token throughput (tok/s):    130.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          4430.14
---------------Time to First Token----------------
Mean TTFT (ms):                          359648.08
Median TTFT (ms):                        359424.54
P99 TTFT (ms):                           714338.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          445.58
Median TPOT (ms):                        462.54
P99 TPOT (ms):                           463.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           441.08
Median ITL (ms):                         448.05
P99 ITL (ms):                            594.59
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  407.05
Total input tokens:                      1997942
Total generated tokens:                  99783
Request throughput (req/s):              1.23
Output token throughput (tok/s):         245.13
Peak output token throughput (tok/s):    1304.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          5153.42
---------------Time to First Token----------------
Mean TTFT (ms):                          197625.03
Median TTFT (ms):                        196621.85
P99 TTFT (ms):                           396056.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          254.61
Median TPOT (ms):                        272.93
P99 TPOT (ms):                           283.50
---------------Inter-token Latency----------------
Mean ITL (ms):                           253.82
Median ITL (ms):                         369.00
P99 ITL (ms):                            377.42
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  175.66
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              2.85
Output token throughput (tok/s):         284.64
Peak output token throughput (tok/s):    1722.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          5967.13
---------------Time to First Token----------------
Mean TTFT (ms):                          86908.30
Median TTFT (ms):                        86575.09
P99 TTFT (ms):                           170607.65
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          301.73
Median TPOT (ms):                        334.17
P99 TPOT (ms):                           334.55
---------------Inter-token Latency----------------
Mean ITL (ms):                           299.28
Median ITL (ms):                         333.99
P99 ITL (ms):                            338.95
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  26.79
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              37.33
Output token throughput (tok/s):         149.33
Peak output token throughput (tok/s):    618.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4916.63
---------------Time to First Token----------------
Mean TTFT (ms):                          19451.46
Median TTFT (ms):                        19150.52
P99 TTFT (ms):                           26523.59
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          233.35
Median TPOT (ms):                        238.61
P99 TPOT (ms):                           251.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           175.01
Median ITL (ms):                         237.82
P99 ITL (ms):                            253.18
==================================================
Скрипт обслуживания после оптимизации 
1
trtllm-serve Qwen/Qwen3-14B --max_seq_len=40960 --enable_chunked_prefill
Результаты бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  727.25
Total input tokens:                      3200000
Total generated tokens:                  9932
Request throughput (req/s):              0.14
Output token throughput (tok/s):         13.66
Peak output token throughput (tok/s):    169.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          4413.82
---------------Time to First Token----------------
Mean TTFT (ms):                          361107.28
Median TTFT (ms):                        361543.94
P99 TTFT (ms):                           716851.40
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          493.18
Median TPOT (ms):                        500.49
P99 TPOT (ms):                           590.17
---------------Inter-token Latency----------------
Mean ITL (ms):                           488.09
Median ITL (ms):                         48.22
P99 ITL (ms):                            2215.91
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  326.81
Total input tokens:                      1997942
Total generated tokens:                  99760
Request throughput (req/s):              1.53
Output token throughput (tok/s):         305.25
Peak output token throughput (tok/s):    1224.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          6418.65
---------------Time to First Token----------------
Mean TTFT (ms):                          155664.65
Median TTFT (ms):                        152609.98
P99 TTFT (ms):                           316584.25
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          196.10
Median TPOT (ms):                        207.95
P99 TPOT (ms):                           218.23
---------------Inter-token Latency----------------
Mean ITL (ms):                           195.11
Median ITL (ms):                         55.84
P99 ITL (ms):                            1038.71
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  140.24
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              3.57
Output token throughput (tok/s):         356.52
Peak output token throughput (tok/s):    2128.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          7474.00
---------------Time to First Token----------------
Mean TTFT (ms):                          67624.26
Median TTFT (ms):                        65786.34
P99 TTFT (ms):                           134519.75
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          313.69
Median TPOT (ms):                        351.99
P99 TPOT (ms):                           365.92
---------------Inter-token Latency----------------
Mean ITL (ms):                           310.55
Median ITL (ms):                         63.96
P99 ITL (ms):                            1002.01
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  15.03
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              66.53
Output token throughput (tok/s):         266.13
Peak output token throughput (tok/s):    883.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          8762.17
---------------Time to First Token----------------
Mean TTFT (ms):                          8166.48
Median TTFT (ms):                        7882.94
P99 TTFT (ms):                           14856.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          715.19
Median TPOT (ms):                        853.12
P99 TPOT (ms):                           1028.67
---------------Inter-token Latency----------------
Mean ITL (ms):                           536.38
Median ITL (ms):                         638.54
P99 ITL (ms):                            1280.94
==================================================