Перейти к содержанию

Оптимизация пропускной способности GLM-4.x на NVIDIA H200 GPU

Вывод

glm-4.6-h200

Рекомендуемая конфигурация для оптимизации пропускной способности GLM-4.x (4.5, 4.6 и 4.7) на H200 GPU:

Команда обслуживания модели 
1
python3 -m sglang.launch_server --model zai-org/GLM-4.6-FP8 --tp 4 --tool-call-parser glm --reasoning-parser glm45

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка базовая конфигурация (vLLM без оптимизации) После оптимизации (в пересчёте на то же число GPU)
ShareGPT Общий TPS: 6030.90
Среднее TPOT(ms): 197.85		 Общий TPS: 7081.34x2 (+134.8%)
Среднее TPOT(ms): 256.43
Короткий запрос Общий TPS: 9497.19
Среднее TPOT(ms): 351.26		 Общий TPS: 21234.81x2 (+347.2%)
Среднее TPOT(ms): 800.25
Запрос средней длины Общий TPS: 17214.85
Среднее TPOT(ms): 269.75		 Общий TPS: 17363.22x2 (+101.7%)
Среднее TPOT(ms): 144.82
Длинный запрос Общий TPS: 16714.91
Среднее TPOT(ms): 205.69		 Общий TPS: 14979.90x2 (+79.2%)
Среднее TPOT(ms): 97.74
Очень длинный запрос Общий TPS: 15066.09
Среднее TPOT(ms): 479.17		 Общий TPS: 12223.53x2 (+62.3%)
Среднее TPOT(ms): 191.61

Примечание

  1. Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
  2. Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
  3. Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.
  4. Квантизация может влиять на точность, однако для большинства моделей FP8 обеспечивает снижение точности менее чем на 1%. Подробности см. в результатах оценки. Поэтому для обслуживания с высокой пропускной способностью настоятельно рекомендуется квантизация FP8.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Цель оптимизации

Высокая пропускная способность при высокой конкурентности запросов.

Условия эксперимента

Модель

zai-org/GLM-4.6

Оборудование

8 × NVIDIA H200 SXM GPU на одном узле.

Версия движка

  • vLLM: v0.11.0
  • SGLang: v0.5.3
  • TensorRT-LLM: v1.0.0

Набор данных бенчмарка

  1. ShareGPT
  2. Случайный набор данных с различной длиной последовательностей:
    • Очень длинный запрос: 32000 входных токенов, 100 выходных токенов
    • Длинный запрос: 4000 входных токенов, 200 выходных токенов
    • Запрос средней длины: 2000 входных токенов, 100 выходных токенов
    • Короткий запрос: 128 входных токенов, 4 выходных токенов

Скрипт бенчмарка

Для оценки производительности модели используется vLLM bench CLI. Бенчмарк запускается следующей командой:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model zai-org/GLM-4.6 --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model zai-org/GLM-4.6 --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Результаты эксперимента

1. Выбор движка инференса

vLLM

Скрипт обслуживания модели 
1
vllm serve zai-org/GLM-4.6 -tp 8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  68.67
Total input tokens:                      214465
Total generated tokens:                  199702
Request throughput (req/s):              14.56
Output token throughput (tok/s):         2907.97
Peak output token throughput (tok/s):    7158.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6030.90
---------------Time to First Token----------------
Mean TTFT (ms):                          15339.53
Median TTFT (ms):                        15377.50
P99 TTFT (ms):                           21078.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          197.85
Median TPOT (ms):                        145.75
P99 TPOT (ms):                           411.50
---------------Inter-token Latency----------------
Mean ITL (ms):                           109.47
Median ITL (ms):                         68.98
P99 ITL (ms):                            413.20
==================================================

SGLang

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.6 --host 0.0.0.0 --port 8000 --tp-size 8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  59.37
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              16.84
Output token throughput (tok/s):         3366.90
Peak output token throughput (tok/s):    7570.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6979.05
---------------Time to First Token----------------
Mean TTFT (ms):                          6044.62
Median TTFT (ms):                        5851.20
P99 TTFT (ms):                           10817.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          282.87
Median TPOT (ms):                        148.97
P99 TPOT (ms):                           1610.70
---------------Inter-token Latency----------------
Mean ITL (ms):                           113.29
Median ITL (ms):                         64.03
P99 ITL (ms):                            369.42
==================================================

Результат: SGLang(6979.05 tok/s) > vLLM (6030.90 tok/s) > TensorRT-LLM (Не поддерживается)

2. Quantization в SGLang

FP8

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.6-FP8 --host 0.0.0.0 --port 8000 --tp-size 8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  52.72
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              18.97
Output token throughput (tok/s):         3791.55
Peak output token throughput (tok/s):    8987.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          7859.29
---------------Time to First Token----------------
Mean TTFT (ms):                          5085.56
Median TTFT (ms):                        4847.02
P99 TTFT (ms):                           10068.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          271.63
Median TPOT (ms):                        144.47
P99 TPOT (ms):                           1542.73
---------------Inter-token Latency----------------
Mean ITL (ms):                           105.34
Median ITL (ms):                         52.76
P99 ITL (ms):                            321.65
==================================================

3. Параллелизм в SGLang

TP8

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.6-FP8 --host 0.0.0.0 --port 8000 --tp-size 8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  52.72
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              18.97
Output token throughput (tok/s):         3791.55
Peak output token throughput (tok/s):    8987.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          7859.29
---------------Time to First Token----------------
Mean TTFT (ms):                          5085.56
Median TTFT (ms):                        4847.02
P99 TTFT (ms):                           10068.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          271.63
Median TPOT (ms):                        144.47
P99 TPOT (ms):                           1542.73
---------------Inter-token Latency----------------
Mean ITL (ms):                           105.34
Median ITL (ms):                         52.76
P99 ITL (ms):                            321.65
==================================================

TP4

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.6-FP8 --host 0.0.0.0 --port 8000 --tp-size 4
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  58.52
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              17.09
Output token throughput (tok/s):         3416.25
Peak output token throughput (tok/s):    7602.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          7081.34
---------------Time to First Token----------------
Mean TTFT (ms):                          5786.78
Median TTFT (ms):                        5769.81
P99 TTFT (ms):                           10134.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          256.43
Median TPOT (ms):                        144.11
P99 TPOT (ms):                           1451.02
---------------Inter-token Latency----------------
Mean ITL (ms):                           109.76
Median ITL (ms):                         62.64
P99 ITL (ms):                            330.68
==================================================

TP4+EP4

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.6-FP8 --host 0.0.0.0 --port 8000 --tp-size 4 --ep-size 4
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  62.06
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              16.11
Output token throughput (tok/s):         3221.11
Peak output token throughput (tok/s):    8209.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6676.84
---------------Time to First Token----------------
Mean TTFT (ms):                          6632.87
Median TTFT (ms):                        6433.53
P99 TTFT (ms):                           11461.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          284.20
Median TPOT (ms):                        157.89
P99 TPOT (ms):                           1629.92
---------------Inter-token Latency----------------
Mean ITL (ms):                           117.77
Median ITL (ms):                         61.25
P99 ITL (ms):                            406.32
==================================================

Сводка методов оптимизации

Метод оптимизации Прирост пропускной способности
Выбор движка +15.7%
Quantization +12.6%
Параллелизм +80.2%

Другие сценарии бенчмарка

Мы дополнительно проверили оптимизированную конфигурацию на разных нагрузках, чтобы оценить её обобщаемость.

Базовый скрипт обслуживания модели 
1
vllm serve zai-org/GLM-4.6 -tp 8
Результаты базового бенчмарка 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  213.02
Total input tokens:                      3200000
Total generated tokens:                  9406
Request throughput (req/s):              0.47
Output token throughput (tok/s):         44.16
Peak output token throughput (tok/s):    489.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          15066.09
---------------Time to First Token----------------
Mean TTFT (ms):                          105089.84
Median TTFT (ms):                        104737.57
P99 TTFT (ms):                           208225.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          479.17
Median TPOT (ms):                        541.92
P99 TPOT (ms):                           544.17
---------------Inter-token Latency----------------
Mean ITL (ms):                           474.58
Median ITL (ms):                         529.63
P99 ITL (ms):                            637.86
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  125.61
Total input tokens:                      1999787
Total generated tokens:                  99801
Request throughput (req/s):              3.98
Output token throughput (tok/s):         794.52
Peak output token throughput (tok/s):    3810.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          16714.91
---------------Time to First Token----------------
Mean TTFT (ms):                          57737.75
Median TTFT (ms):                        59379.10
P99 TTFT (ms):                           116713.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          205.69
Median TPOT (ms):                        241.12
P99 TPOT (ms):                           260.89
---------------Inter-token Latency----------------
Mean ITL (ms):                           205.05
Median ITL (ms):                         59.67
P99 ITL (ms):                            424.92
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  60.96
Total input tokens:                      999429
Total generated tokens:                  50000
Request throughput (req/s):              8.20
Output token throughput (tok/s):         820.20
Peak output token throughput (tok/s):    5415.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          17214.85
---------------Time to First Token----------------
Mean TTFT (ms):                          30215.66
Median TTFT (ms):                        29924.42
P99 TTFT (ms):                           55334.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          269.75
Median TPOT (ms):                        283.63
P99 TPOT (ms):                           407.32
---------------Inter-token Latency----------------
Mean ITL (ms):                           267.55
Median ITL (ms):                         402.36
P99 ITL (ms):                            417.66
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  13.89
Total input tokens:                      127881
Total generated tokens:                  4000
Request throughput (req/s):              72.01
Output token throughput (tok/s):         288.05
Peak output token throughput (tok/s):    2063.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          9497.19
---------------Time to First Token----------------
Mean TTFT (ms):                          11467.27
Median TTFT (ms):                        10707.53
P99 TTFT (ms):                           13722.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          351.26
Median TPOT (ms):                        397.14
P99 TPOT (ms):                           409.34
---------------Inter-token Latency----------------
Mean ITL (ms):                           263.45
Median ITL (ms):                         389.46
P99 ITL (ms):                            443.88
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  103.11
Total input tokens:                      22992
Total generated tokens:                  21794
Request throughput (req/s):              0.97
Output token throughput (tok/s):         211.38
Peak output token throughput (tok/s):    232.00
Peak concurrent requests:                9.00
Total Token throughput (tok/s):          434.37
---------------Time to First Token----------------
Mean TTFT (ms):                          70.16
Median TTFT (ms):                        62.63
P99 TTFT (ms):                           148.47
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          18.11
Median TPOT (ms):                        17.80
P99 TPOT (ms):                           23.44
---------------Inter-token Latency----------------
Mean ITL (ms):                           17.85
Median ITL (ms):                         17.55
P99 ITL (ms):                            26.92
==================================================
Скрипт обслуживания после оптимизации 
1
2
python3 -m sglang.launch_server --model zai-org/GLM-4.6-FP8 --tp 4 \
    --tool-call-parser glm --reasoning-parser glm45
Результаты бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  262.58
Total input tokens:                      3200000
Total generated tokens:                  9703
Request throughput (req/s):              0.38
Output token throughput (tok/s):         36.95
Peak output token throughput (tok/s):    390.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          12223.53
---------------Time to First Token----------------
Mean TTFT (ms):                          129699.21
Median TTFT (ms):                        129839.73
P99 TTFT (ms):                           257123.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          191.61
Median TPOT (ms):                        181.54
P99 TPOT (ms):                           350.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           189.68
Median ITL (ms):                         40.53
P99 ITL (ms):                            71.16
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  140.16
Total input tokens:                      1999787
Total generated tokens:                  99801
Request throughput (req/s):              3.57
Output token throughput (tok/s):         712.05
Peak output token throughput (tok/s):    2688.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          14979.90
---------------Time to First Token----------------
Mean TTFT (ms):                          65196.25
Median TTFT (ms):                        67840.59
P99 TTFT (ms):                           131861.02
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          97.74
Median TPOT (ms):                        94.74
P99 TPOT (ms):                           149.34
---------------Inter-token Latency----------------
Mean ITL (ms):                           97.55
Median ITL (ms):                         47.60
P99 ITL (ms):                            314.57
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  60.44
Total input tokens:                      999429
Total generated tokens:                  50000
Request throughput (req/s):              8.27
Output token throughput (tok/s):         827.27
Peak output token throughput (tok/s):    4460.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          17363.22
---------------Time to First Token----------------
Mean TTFT (ms):                          27600.49
Median TTFT (ms):                        29627.26
P99 TTFT (ms):                           56644.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          144.82
Median TPOT (ms):                        142.76
P99 TPOT (ms):                           250.18
---------------Inter-token Latency----------------
Mean ITL (ms):                           143.46
Median ITL (ms):                         57.80
P99 ITL (ms):                            145.06
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  6.21
Total input tokens:                      127881
Total generated tokens:                  4000
Request throughput (req/s):              161.01
Output token throughput (tok/s):         644.06
Peak output token throughput (tok/s):    1924.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          21234.81
---------------Time to First Token----------------
Mean TTFT (ms):                          3551.50
Median TTFT (ms):                        3313.50
P99 TTFT (ms):                           5528.41
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          800.25
Median TPOT (ms):                        936.39
P99 TPOT (ms):                           1413.56
---------------Inter-token Latency----------------
Mean ITL (ms):                           600.19
Median ITL (ms):                         0.01
P99 ITL (ms):                            3985.12
==================================================