Перейти к содержанию

Оптимизация пропускной способности GLM-4.5-Air на NVIDIA H100 GPU

Вывод

glm-4.5-air-h100

Рекомендуемая конфигурация для оптимизации пропускной способности GLM-4.5-Air на H100 GPU:

Команда обслуживания модели 
1
2
python3 -m sglang.launch_server --model zai-org/GLM-4.5-Air-FP8 \
--tp-size 4 --ep-size 4 --tool-call-parser glm --reasoning-parser glm45

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка базовая конфигурация (vLLM без оптимизации) После оптимизации (в пересчёте на то же число GPU)
ShareGPT Общий TPS: 7909.53
Среднее TPOT(ms): 134.47		 Общий TPS: 12411.25 (+56.9%)
Среднее TPOT(ms): 153.80
Короткий запрос Общий TPS: 7066.67
Среднее TPOT(ms): 214.38		 Общий TPS: 28621.61 (+305.0%)
Среднее TPOT(ms): 327.73
Запрос средней длины Общий TPS: 22582.76
Среднее TPOT(ms): 125.44		 Общий TPS: 35741.37 (+58.3%)
Среднее TPOT(ms): 106.94
Длинный запрос Общий TPS: 21705.53
Среднее TPOT(ms): 82.53		 Общий TPS: 33207.02 (+53.0%)
Среднее TPOT(ms): 71.49
Очень длинный запрос Общий TPS: 20732.85
Среднее TPOT(ms): 176.99		 Общий TPS: 26683.88 (+28.7%)
Среднее TPOT(ms): 200.08

Примечание

  1. Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
  2. Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
  3. Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.
  4. Квантизация может влиять на точность, однако для большинства моделей FP8 обеспечивает снижение точности менее чем на 1%. Подробности см. в результатах оценки. Поэтому для обслуживания с высокой пропускной способностью настоятельно рекомендуется квантизация FP8.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Цель оптимизации

Высокая пропускная способность при высокой конкурентности запросов.

Условия эксперимента

Модель

zai-org/GLM-4.5-Air

Оборудование

4 × NVIDIA H100 SXM GPU на одном узле.

Версия движка

  • vLLM: v0.11.0
  • SGLang: v0.5.4.post2
  • TensorRT-LLM: v1.2.0rc1

Набор данных бенчмарка

  1. ShareGPT
  2. Случайный набор данных с различной длиной последовательностей:
    • Очень длинный запрос: 32000 входных токенов, 100 выходных токенов
    • Длинный запрос: 4000 входных токенов, 200 выходных токенов
    • Запрос средней длины: 2000 входных токенов, 100 выходных токенов
    • Короткий запрос: 128 входных токенов, 4 выходных токенов

Скрипт бенчмарка

Для оценки производительности модели используется vLLM bench CLI. Бенчмарк запускается следующей командой:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model zai-org/GLM-4.5-Air --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model zai-org/GLM-4.5-Air --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Результаты эксперимента

1. Выбор движка инференса

vLLM

Скрипт обслуживания модели 
1
vllm serve zai-org/GLM-4.5-Air -tp 4
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  52.39
Total input tokens:                      214465
Total generated tokens:                  199895
Request throughput (req/s):              19.09
Output token throughput (tok/s):         3815.71
Peak output token throughput (tok/s):    10558.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          7909.53
---------------Time to First Token----------------
Mean TTFT (ms):                          16383.88
Median TTFT (ms):                        16164.27
P99 TTFT (ms):                           19975.18
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          134.47
Median TPOT (ms):                        105.53
P99 TPOT (ms):                           276.78
---------------Inter-token Latency----------------
Mean ITL (ms):                           74.75
Median ITL (ms):                         47.07
P99 ITL (ms):                            265.69
==================================================

SGLang

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air --tp-size 4
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  44.10
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              22.67
Output token throughput (tok/s):         4532.66
Peak output token throughput (tok/s):    12110.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          9395.49
---------------Time to First Token----------------
Mean TTFT (ms):                          4371.21
Median TTFT (ms):                        4175.32
P99 TTFT (ms):                           8363.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          222.86
Median TPOT (ms):                        132.57
P99 TPOT (ms):                           1165.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           91.03
Median ITL (ms):                         40.24
P99 ITL (ms):                            525.39
==================================================

Результат: SGLang(9395.49 tok/s) > vLLM (7909.53 tok/s) > TensorRT-LLM (Не поддерживается)

2. Quantization в SGLang

FP8

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model zai-org/GLM-4.5-Air-FP8 --tp-size 4
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  34.54
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              28.95
Output token throughput (tok/s):         5787.27
Peak output token throughput (tok/s):    20354.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          11996.08
---------------Time to First Token----------------
Mean TTFT (ms):                          4436.82
Median TTFT (ms):                        4318.99
P99 TTFT (ms):                           6248.30
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          179.03
Median TPOT (ms):                        104.28
P99 TPOT (ms):                           806.18
---------------Inter-token Latency----------------
Mean ITL (ms):                           72.38
Median ITL (ms):                         25.48
P99 ITL (ms):                            438.37
==================================================

3. Параллелизм в SGLang

TP4+EP4

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model zai-org/GLM-4.5-Air --tp-size 4 --ep-size 4
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  33.39
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              29.95
Output token throughput (tok/s):         5987.56
Peak output token throughput (tok/s):    17690.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          12411.25
---------------Time to First Token----------------
Mean TTFT (ms):                          4086.62
Median TTFT (ms):                        4050.40
P99 TTFT (ms):                           6097.74
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          153.80
Median TPOT (ms):                        105.20
P99 TPOT (ms):                           721.74
---------------Inter-token Latency----------------
Mean ITL (ms):                           67.59
Median ITL (ms):                         26.23
P99 ITL (ms):                            378.32
==================================================

TP2

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air  --tp-size 2
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  66.93
Total input tokens:                      214465
Total generated tokens:                  199824
Request throughput (req/s):              14.94
Output token throughput (tok/s):         2985.45
Peak output token throughput (tok/s):    6512.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6189.64
---------------Time to First Token----------------
Mean TTFT (ms):                          12314.75
Median TTFT (ms):                        4776.06
P99 TTFT (ms):                           44680.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          145.35
Median TPOT (ms):                        142.17
P99 TPOT (ms):                           377.52
---------------Inter-token Latency----------------
Mean ITL (ms):                           113.40
Median ITL (ms):                         93.47
P99 ITL (ms):                            333.86
==================================================

TP2+EP2

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air  --tp-size 2 --ep-size 2
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  71.98
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              13.89
Output token throughput (tok/s):         2777.39
Peak output token throughput (tok/s):    6953.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5757.08
---------------Time to First Token----------------
Mean TTFT (ms):                          14606.79
Median TTFT (ms):                        7321.02
P99 TTFT (ms):                           50010.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          158.73
Median TPOT (ms):                        150.07
P99 TPOT (ms):                           394.02
---------------Inter-token Latency----------------
Mean ITL (ms):                           119.76
Median ITL (ms):                         99.62
P99 ITL (ms):                            266.91
==================================================

PP4

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.5-Air  --pp-size 4
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  118.50
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              8.44
Output token throughput (tok/s):         1686.89
Peak output token throughput (tok/s):    5980.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          3496.65
---------------Time to First Token----------------
Mean TTFT (ms):                          4772.90
Median TTFT (ms):                        4737.52
P99 TTFT (ms):                           6786.89
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          243.24
Median TPOT (ms):                        187.96
P99 TPOT (ms):                           872.62
---------------Inter-token Latency----------------
Mean ITL (ms):                           173.87
Median ITL (ms):                         170.67
P99 ITL (ms):                            227.82
==================================================

Сводка методов оптимизации

Метод оптимизации Прирост пропускной способности
Выбор движка +18.8%
Quantization +27.7%
Параллелизм +3.5%

Другие сценарии бенчмарка

Мы дополнительно проверили оптимизированную конфигурацию на разных нагрузках, чтобы оценить её обобщаемость.

Базовый скрипт обслуживания модели 
1
vllm serve zai-org/GLM-4.5-Air -tp 4
Результаты базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  498.76
Total input tokens:                      3200000
Total generated tokens:                  9987
Request throughput (req/s):              0.20
Output token throughput (tok/s):         20.02
Peak output token throughput (tok/s):    218.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          6435.97
---------------Time to First Token----------------
Mean TTFT (ms):                          248013.29
Median TTFT (ms):                        248015.97
P99 TTFT (ms):                           491884.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          305.47
Median TPOT (ms):                        317.93
P99 TPOT (ms):                           319.14
---------------Inter-token Latency----------------
Mean ITL (ms):                           303.40
Median ITL (ms):                         309.84
P99 ITL (ms):                            406.26
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  300.98
Total input tokens:                      1999787
Total generated tokens:                  99547
Request throughput (req/s):              1.66
Output token throughput (tok/s):         330.75
Peak output token throughput (tok/s):    1743.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          6975.10
---------------Time to First Token----------------
Mean TTFT (ms):                          142573.31
Median TTFT (ms):                        141422.49
P99 TTFT (ms):                           289825.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          267.38
Median TPOT (ms):                        295.72
P99 TPOT (ms):                           300.72
---------------Inter-token Latency----------------
Mean ITL (ms):                           266.48
Median ITL (ms):                         295.37
P99 ITL (ms):                            307.18
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  121.41
Total input tokens:                      999429
Total generated tokens:                  50000
Request throughput (req/s):              4.12
Output token throughput (tok/s):         411.83
Peak output token throughput (tok/s):    1796.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          8643.75
---------------Time to First Token----------------
Mean TTFT (ms):                          60300.00
Median TTFT (ms):                        60039.46
P99 TTFT (ms):                           116422.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          204.31
Median TPOT (ms):                        224.99
P99 TPOT (ms):                           225.56
---------------Inter-token Latency----------------
Mean ITL (ms):                           202.57
Median ITL (ms):                         224.72
P99 ITL (ms):                            230.94
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  21.03
Total input tokens:                      127881
Total generated tokens:                  4000
Request throughput (req/s):              47.55
Output token throughput (tok/s):         190.21
Peak output token throughput (tok/s):    952.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6271.24
---------------Time to First Token----------------
Mean TTFT (ms):                          16583.77
Median TTFT (ms):                        16009.09
P99 TTFT (ms):                           20726.61
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          153.00
Median TPOT (ms):                        153.83
P99 TPOT (ms):                           173.39
---------------Inter-token Latency----------------
Mean ITL (ms):                           114.74
Median ITL (ms):                         153.44
P99 ITL (ms):                            192.15
==================================================
Скрипт обслуживания после оптимизации 
1
2
python3 -m sglang.launch_server --model GLM-4.5-Air --tp-size 4 --ep-size 4 \
    --tool-call-parser glm --reasoning-parser glm45
Результаты бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  403.99
Total input tokens:                      3200000
Total generated tokens:                  9931
Request throughput (req/s):              0.25
Output token throughput (tok/s):         24.58
Peak output token throughput (tok/s):    332.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          7945.66
---------------Time to First Token----------------
Mean TTFT (ms):                          200446.56
Median TTFT (ms):                        199894.57
P99 TTFT (ms):                           397275.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          276.41
Median TPOT (ms):                        295.02
P99 TPOT (ms):                           410.95
---------------Inter-token Latency----------------
Mean ITL (ms):                           271.13
Median ITL (ms):                         32.39
P99 ITL (ms):                            3555.22
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  230.67
Total input tokens:                      1999787
Total generated tokens:                  99437
Request throughput (req/s):              2.17
Output token throughput (tok/s):         431.09
Peak output token throughput (tok/s):    1680.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          9100.71
---------------Time to First Token----------------
Mean TTFT (ms):                          109316.41
Median TTFT (ms):                        113614.39
P99 TTFT (ms):                           222368.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          124.68
Median TPOT (ms):                        124.45
P99 TPOT (ms):                           179.33
---------------Inter-token Latency----------------
Mean ITL (ms):                           123.34
Median ITL (ms):                         59.70
P99 ITL (ms):                            294.40
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  95.78
Total input tokens:                      999429
Total generated tokens:                  49964
Request throughput (req/s):              5.22
Output token throughput (tok/s):         521.66
Peak output token throughput (tok/s):    3160.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          10956.49
---------------Time to First Token----------------
Mean TTFT (ms):                          45224.78
Median TTFT (ms):                        44492.00
P99 TTFT (ms):                           91909.36
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          165.34
Median TPOT (ms):                        163.74
P99 TPOT (ms):                           281.83
---------------Inter-token Latency----------------
Mean ITL (ms):                           162.06
Median ITL (ms):                         63.35
P99 ITL (ms):                            166.19
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  10.14
Total input tokens:                      127881
Total generated tokens:                  4000
Request throughput (req/s):              98.67
Output token throughput (tok/s):         394.66
Peak output token throughput (tok/s):    2166.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          13012.20
---------------Time to First Token----------------
Mean TTFT (ms):                          5882.75
Median TTFT (ms):                        5678.55
P99 TTFT (ms):                           9321.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1246.57
Median TPOT (ms):                        1420.02
P99 TPOT (ms):                           2379.65
---------------Inter-token Latency----------------
Mean ITL (ms):                           747.94
Median ITL (ms):                         59.22
P99 ITL (ms):                            6782.02
==================================================