Перейти к содержанию

Оптимизация пропускной способности GLM-5.2 на H200

Вывод

Результат оптимизации пропускной способности

Рекомендуемая конфигурация для оптимизации пропускной способности zai-org/GLM-5.2-FP8 на NVIDIA H200x8:

Команда обслуживания модели 
1
2
3
4
5
6
7
8
9
python3 -m sglang.launch_server \
    --model-path zai-org/GLM-5.2-FP8 \
    --tp-size=8 \
    --dp-size=8 \
    --enable-dp-attention \
    --mem-fraction-static=0.85 \
    --moe-a2a-backend=deepep \
    --max-running-requests=256 \
    --kv-cache-dtype=fp8_e4m3

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка базовая конфигурация (vLLM без оптимизации) После оптимизации
Balanced: 8K входных / 1K выходных (цель оптимизации) Общий TPS: 5481.90
Среднее TPOT(ms): 636.24		 Общий TPS: 17525.76 (+219.70%)
Среднее TPOT(ms): 217.89
Короткий контекст (1K входных / 128 выходных) Общий TPS: 13196.90
Среднее TPOT(ms): 316.73		 Общий TPS: 18276.60 (+38.49%)
Среднее TPOT(ms): 211.06
Длинный контекст (32K входных / 100 выходных) Общий TPS: 10513.68
Среднее TPOT(ms): 1320.02		 Общий TPS: 26433.15 (+151.42%)
Среднее TPOT(ms): 537.26
Интенсивная генерация (1K входных / 2K выходных) Общий TPS: 2268.18
Среднее TPOT(ms): 42.11		 Общий TPS: 2267.33 (-0.04%)
Среднее TPOT(ms): 46.54
ShareGPT Общий TPS: 6040.94
Среднее TPOT(ms): 149.24		 Общий TPS: 2845.48 (-52.90%)
Среднее TPOT(ms): 331.21

Примечание

  1. Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
  2. Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
  3. Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.
  4. Квантизация может влиять на точность, однако для большинства моделей FP8 обеспечивает снижение точности менее чем на 1%. Подробности см. в результатах оценки. Поэтому для обслуживания с низкой задержкой настоятельно рекомендуется квантизация FP8.
  5. Рекомендуемая конфигурация настроена на высокую конкурентность и входы средней или большой длины (опорная нагрузка: 8K входных / 1K выходных токенов). В диалоговых нагрузках с короткими входами и выходами, таких как ShareGPT, она уступает обычному тензорному параллелизму: оптимизации DP-attention и DeepEP рассчитаны на длинные последовательности, а короткие запросы не компенсируют их накладные расходы (см. ShareGPT в разделе других бенчмарков). Этот компромисс подходит целевому применению GLM-5.2: сильные возможности программирования делают обслуживание агентов разработки основным сценарием, где контекст преимущественно длинный — именно на него и рассчитана эта конфигурация.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Условия эксперимента

Модель

zai-org/GLM-5.2-FP8

Оборудование

NVIDIA H200x8

Версия движка

  • vLLM v0.24.0
  • SGLang v0.5.14

Методика бенчмарка

В этом проекте используется функция запуска бенчмарка GPUStack одним щелчком для нагрузок обслуживания моделей. Бенчмарки этого документа выполнялись именно таким способом.

Бенчмарки GPUStack реализованы на основе guidellm через проект-обёртку benchmark-runner.

Для перечисленных ниже конфигураций GPUStack выполняет развёртывание модели, отправку заданий бенчмарка и сбор результатов.

Профили бенчмарка

Balanced
1
2
3
4
5
6
dataset_name: Random
dataset_input_tokens: 8192
dataset_output_tokens: 1024
dataset_seed: 42
request_rate: 1000
total_requests: 1000
Короткий контекст
1
2
3
4
5
6
dataset_name: Random
dataset_input_tokens: 1024
dataset_output_tokens: 128
dataset_seed: 42
request_rate: 1000
total_requests: 1000
Длинный контекст
1
2
3
4
5
6
dataset_name: Random
dataset_input_tokens: 32000
dataset_output_tokens: 100
dataset_seed: 42
request_rate: 1
total_requests: 100
Интенсивная генерация
1
2
3
4
5
6
dataset_name: Random
dataset_input_tokens: 1000
dataset_output_tokens: 2000
dataset_seed: 42
request_rate: 1
total_requests: 200
ShareGPT
1
2
3
dataset_name: ShareGPT
request_rate: 1000
total_requests: 1000

Результаты эксперимента

Выбор движка инференса

vLLM

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
    --tensor-parallel-size=8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  1684.76
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              0.59
Output token throughput (tok/s):         608.31
Peak output token throughput (tok/s):    21900.04
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          5481.90
----------------------Latency---------------------
Mean Latency(s):                          651.51
Median Latency(s):                        823.57
P95 Latency(s):                           857.92
P99 Latency(s):                           858.57
---------------Time to First Token----------------
Mean TTFT (ms):                          592315.54
Median TTFT (ms):                        762631.30
P95 TTFT (ms):                           799738.92
P99 TTFT (ms):                           800277.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          636.24
Median TPOT (ms):                        804.27
P95 TPOT (ms):                           837.81
P99 TPOT (ms):                           838.45
---------------Inter-token Latency----------------
Mean ITL (ms):                           57.86
Median ITL (ms):                         56.85
P95 ITL (ms):                            69.41
P99 ITL (ms):                            90.01
==================================================

SGLang

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
    --tp-size=8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  1425.58
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              0.70
Output token throughput (tok/s):         719.90
Peak output token throughput (tok/s):    36770.60
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          6487.53
----------------------Latency---------------------
Mean Latency(s):                          553.77
Median Latency(s):                        674.98
P95 Latency(s):                           738.28
P99 Latency(s):                           1087.06
---------------Time to First Token----------------
Mean TTFT (ms):                          499353.08
Median TTFT (ms):                        640537.31
P95 TTFT (ms):                           692823.09
P99 TTFT (ms):                           694859.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          540.79
Median TPOT (ms):                        659.16
P95 TPOT (ms):                           720.97
P99 TPOT (ms):                           1061.58
---------------Inter-token Latency----------------
Mean ITL (ms):                           53.19
Median ITL (ms):                         41.87
P95 ITL (ms):                            53.14
P99 ITL (ms):                            661.38
==================================================
  • Итог: SGLang Общий TPS = 6487.53, vLLM Общий TPS = 5481.90. SGLang быстрее на 1005.62 tok/s (18.34%); Среднее TPOT = 540.79 ms против 636.24 ms, снижение на 95.45 ms (15.00%).

Стратегия параллелизма

TP

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
    --tp-size=8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  1425.58
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              0.70
Output token throughput (tok/s):         719.90
Peak output token throughput (tok/s):    36770.60
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          6487.53
----------------------Latency---------------------
Mean Latency(s):                          553.77
Median Latency(s):                        674.98
P95 Latency(s):                           738.28
P99 Latency(s):                           1087.06
---------------Time to First Token----------------
Mean TTFT (ms):                          499353.08
Median TTFT (ms):                        640537.31
P95 TTFT (ms):                           692823.09
P99 TTFT (ms):                           694859.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          540.79
Median TPOT (ms):                        659.16
P95 TPOT (ms):                           720.97
P99 TPOT (ms):                           1061.58
---------------Inter-token Latency----------------
Mean ITL (ms):                           53.19
Median ITL (ms):                         41.87
P95 ITL (ms):                            53.14
P99 ITL (ms):                            661.38
==================================================

TP + DP-Attention

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
    --tp-size=8
--enable-dp-attention
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  1464.79
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              0.68
Output token throughput (tok/s):         707.03
Peak output token throughput (tok/s):    36792.16
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          6371.59
----------------------Latency---------------------
Mean Latency(s):                          566.77
Median Latency(s):                        691.84
P95 Latency(s):                           758.76
P99 Latency(s):                           1226.15
---------------Time to First Token----------------
Mean TTFT (ms):                          508420.96
Median TTFT (ms):                        649948.91
P95 TTFT (ms):                           704806.45
P99 TTFT (ms):                           709287.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          553.48
Median TPOT (ms):                        675.62
P95 TPOT (ms):                           740.98
P99 TPOT (ms):                           1197.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           57.03
Median ITL (ms):                         40.44
P95 ITL (ms):                            52.07
P99 ITL (ms):                            658.59
==================================================

TP + DP + DP-Attention

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
    --tp-size=8
--dp-size=8
--enable-dp-attention
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  1334.85
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              0.75
Output token throughput (tok/s):         778.01
Peak output token throughput (tok/s):    33009.89
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          7011.17
----------------------Latency---------------------
Mean Latency(s):                          509.42
Median Latency(s):                        662.31
P95 Latency(s):                           667.58
P99 Latency(s):                           667.92
---------------Time to First Token----------------
Mean TTFT (ms):                          474839.81
Median TTFT (ms):                        625093.03
P95 TTFT (ms):                           635816.93
P99 TTFT (ms):                           636463.47
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          497.48
Median TPOT (ms):                        646.79
P95 TPOT (ms):                           651.93
P99 TPOT (ms):                           652.26
---------------Inter-token Latency----------------
Mean ITL (ms):                           33.81
Median ITL (ms):                         33.03
P95 ITL (ms):                            37.77
P99 ITL (ms):                            38.03
==================================================

TP + DP + DP-Attention + EP

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
    --tp-size=8
--enable-dp-attention
--dp-size=8
--ep-size=8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  1430.76
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              0.70
Output token throughput (tok/s):         724.90
Peak output token throughput (tok/s):    36964.00
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          6532.56
----------------------Latency---------------------
Mean Latency(s):                          546.29
Median Latency(s):                        710.24
P95 Latency(s):                           716.16
P99 Latency(s):                           716.23
---------------Time to First Token----------------
Mean TTFT (ms):                          509145.51
Median TTFT (ms):                        672291.77
P95 TTFT (ms):                           682441.07
P99 TTFT (ms):                           682611.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          533.49
Median TPOT (ms):                        693.59
P95 TPOT (ms):                           699.37
P99 TPOT (ms):                           699.44
---------------Inter-token Latency----------------
Mean ITL (ms):                           36.31
Median ITL (ms):                         35.36
P95 ITL (ms):                            40.52
P99 ITL (ms):                            40.69
==================================================
  • Итог: TP + DP + DP-Attention Общий TPS = 7011.17, TP + DP-Attention Общий TPS = 6371.59. TP + DP + DP-Attention быстрее на 639.58 tok/s (10.04%); Среднее TPOT = 497.48 ms против 553.48 ms, снижение на 56.00 ms (10.12%).

Доля памяти

auto

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
    --tp-size=8
--dp-size=8
--enable-dp-attention
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  1587.01
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              0.63
Output token throughput (tok/s):         652.68
Peak output token throughput (tok/s):    28960.54
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          5881.79
----------------------Latency---------------------
Mean Latency(s):                          608.53
Median Latency(s):                        792.41
P95 Latency(s):                           798.47
P99 Latency(s):                           798.63
---------------Time to First Token----------------
Mean TTFT (ms):                          565850.91
Median TTFT (ms):                        749005.91
P95 TTFT (ms):                           758437.70
P99 TTFT (ms):                           759022.89
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          594.26
Median TPOT (ms):                        773.84
P95 TPOT (ms):                           779.76
P99 TPOT (ms):                           779.91
---------------Inter-token Latency----------------
Mean ITL (ms):                           41.71
Median ITL (ms):                         41.02
P95 ITL (ms):                            45.77
P99 ITL (ms):                            45.99
==================================================

0.8

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  890.94
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.12
Output token throughput (tok/s):         1173.22
Peak output token throughput (tok/s):    36646.25
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          10572.77
----------------------Latency---------------------
Mean Latency(s):                          352.36
Median Latency(s):                        385.98
P95 Latency(s):                           453.81
P99 Latency(s):                           743.15
---------------Time to First Token----------------
Mean TTFT (ms):                          283237.85
Median TTFT (ms):                        338538.99
P95 TTFT (ms):                           395476.65
P99 TTFT (ms):                           398036.19
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          344.10
Median TPOT (ms):                        376.93
P95 TPOT (ms):                           443.17
P99 TPOT (ms):                           725.73
---------------Inter-token Latency----------------
Mean ITL (ms):                           67.57
Median ITL (ms):                         52.23
P95 ITL (ms):                            61.80
P99 ITL (ms):                            400.53
==================================================

0.85

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  688.60
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.45
Output token throughput (tok/s):         1527.01
Peak output token throughput (tok/s):    46500.08
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          13760.98
----------------------Latency---------------------
Mean Latency(s):                          273.29
Median Latency(s):                        272.67
P95 Latency(s):                           362.52
P99 Latency(s):                           488.73
---------------Time to First Token----------------
Mean TTFT (ms):                          201642.60
Median TTFT (ms):                        219374.83
P95 TTFT (ms):                           293139.84
P99 TTFT (ms):                           297529.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          266.89
Median TPOT (ms):                        266.28
P95 TPOT (ms):                           354.02
P99 TPOT (ms):                           477.28
---------------Inter-token Latency----------------
Mean ITL (ms):                           70.04
Median ITL (ms):                         65.72
P95 ITL (ms):                            85.73
P99 ITL (ms):                            276.51
==================================================
  • Итог: 0.85 Общий TPS = 13760.98, auto Общий TPS = 5881.79. 0.85 быстрее на 7879.19 tok/s (133.96%); Среднее TPOT = 266.89 ms против 594.26 ms, снижение на 327.38 ms (55.09%).

Размер блока prefill

По умолчанию (default)

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  688.60
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.45
Output token throughput (tok/s):         1527.01
Peak output token throughput (tok/s):    46500.08
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          13760.98
----------------------Latency---------------------
Mean Latency(s):                          273.29
Median Latency(s):                        272.67
P95 Latency(s):                           362.52
P99 Latency(s):                           488.73
---------------Time to First Token----------------
Mean TTFT (ms):                          201642.60
Median TTFT (ms):                        219374.83
P95 TTFT (ms):                           293139.84
P99 TTFT (ms):                           297529.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          266.89
Median TPOT (ms):                        266.28
P95 TPOT (ms):                           354.02
P99 TPOT (ms):                           477.28
---------------Inter-token Latency----------------
Mean ITL (ms):                           70.04
Median ITL (ms):                         65.72
P95 ITL (ms):                            85.73
P99 ITL (ms):                            276.51
==================================================

8K

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
5
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--chunked-prefill-size=8192
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  689.11
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.45
Output token throughput (tok/s):         1526.75
Peak output token throughput (tok/s):    52394.68
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          13758.60
----------------------Latency---------------------
Mean Latency(s):                          273.27
Median Latency(s):                        272.75
P95 Latency(s):                           362.79
P99 Latency(s):                           489.22
---------------Time to First Token----------------
Mean TTFT (ms):                          201640.48
Median TTFT (ms):                        219319.19
P95 TTFT (ms):                           293076.86
P99 TTFT (ms):                           297429.96
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          266.86
Median TPOT (ms):                        266.36
P95 TPOT (ms):                           354.28
P99 TPOT (ms):                           477.76
---------------Inter-token Latency----------------
Mean ITL (ms):                           70.02
Median ITL (ms):                         65.70
P95 ITL (ms):                            85.80
P99 ITL (ms):                            276.29
==================================================
  • Итог: default Общий TPS = 13760.98, 8K Общий TPS = 13758.60. default быстрее на 2.37 tok/s (0.02%); Среднее TPOT = 266.89 ms против 266.86 ms, увеличение на 0.03 ms (0.01% медленнее).

Нехватка памяти при большом блоке prefill

Установка --chunked-prefill-size в 16384 или 32768 вызывает сбой из-за нехватки памяти в этой конфигурации. Больший размер блока увеличивает резерв для активаций, который планировщик памяти вычитает из бюджета KV-кэша; оставшейся памяти недостаточно для весов FP8 размера ~700B вместе с KV-кэшем. Работают только 8192 и значение по умолчанию; различие между ними находится в пределах шума. При входе 8K размер блока prefill не даёт прироста пропускной способности, поэтому в рекомендуемой конфигурации оставлен по умолчанию.

DeepEP и максимум одновременных запросов

max-running 64

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
5
6
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--moe-a2a-backend=deepep
--max-running-requests=64
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  945.02
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.05
Output token throughput (tok/s):         1105.14
Peak output token throughput (tok/s):    49070.69
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          9959.18
----------------------Latency---------------------
Mean Latency(s):                          367.92
Median Latency(s):                        462.73
P95 Latency(s):                           516.66
P99 Latency(s):                           517.52
---------------Time to First Token----------------
Mean TTFT (ms):                          325930.74
Median TTFT (ms):                        419854.01
P95 TTFT (ms):                           468446.97
P99 TTFT (ms):                           469030.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          359.30
Median TPOT (ms):                        451.88
P95 TPOT (ms):                           504.55
P99 TPOT (ms):                           505.39
---------------Inter-token Latency----------------
Mean ITL (ms):                           41.05
Median ITL (ms):                         41.00
P95 ITL (ms):                            47.87
P99 ITL (ms):                            48.31
==================================================

max-running 256

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
5
6
    --tp-size=8
--dp-size=8
--enable-dp-attention
--moe-a2a-backend=deepep
--mem-fraction-static=0.85
--max-running-requests=256
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  657.33
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.52
Output token throughput (tok/s):         1603.58
Peak output token throughput (tok/s):    53623.54
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          14451.05
----------------------Latency---------------------
Mean Latency(s):                          261.76
Median Latency(s):                        288.37
P95 Latency(s):                           362.40
P99 Latency(s):                           474.82
---------------Time to First Token----------------
Mean TTFT (ms):                          200415.03
Median TTFT (ms):                        235618.38
P95 TTFT (ms):                           296378.61
P99 TTFT (ms):                           299199.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          255.63
Median TPOT (ms):                        281.61
P95 TPOT (ms):                           353.91
P99 TPOT (ms):                           463.69
---------------Inter-token Latency----------------
Mean ITL (ms):                           59.97
Median ITL (ms):                         53.70
P95 ITL (ms):                            70.44
P99 ITL (ms):                            286.92
==================================================

max-running 512

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
5
6
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--moe-a2a-backend=deepep
--max-running-requests=512
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  666.14
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.50
Output token throughput (tok/s):         1581.39
Peak output token throughput (tok/s):    38164.10
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          14251.08
----------------------Latency---------------------
Mean Latency(s):                          265.47
Median Latency(s):                        292.45
P95 Latency(s):                           368.40
P99 Latency(s):                           482.40
---------------Time to First Token----------------
Mean TTFT (ms):                          203755.52
Median TTFT (ms):                        239495.62
P95 TTFT (ms):                           301525.04
P99 TTFT (ms):                           304478.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          259.25
Median TPOT (ms):                        285.59
P95 TPOT (ms):                           359.76
P99 TPOT (ms):                           471.09
---------------Inter-token Latency----------------
Mean ITL (ms):                           60.33
Median ITL (ms):                         54.22
P95 ITL (ms):                            70.59
P99 ITL (ms):                            291.22
==================================================
  • Итог: max-running 256 Общий TPS = 14451.05, max-running 64 Общий TPS = 9959.18. max-running 256 быстрее на 4491.87 tok/s (45.10%); Среднее TPOT = 255.63 ms против 359.30 ms, снижение на 103.67 ms (28.85%).

Режим DeepEP

auto

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
5
6
    --tp-size=8
--dp-size=8
--enable-dp-attention
--moe-a2a-backend=deepep
--mem-fraction-static=0.85
--max-running-requests=256
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  657.33
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.52
Output token throughput (tok/s):         1603.58
Peak output token throughput (tok/s):    53623.54
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          14451.05
----------------------Latency---------------------
Mean Latency(s):                          261.76
Median Latency(s):                        288.37
P95 Latency(s):                           362.40
P99 Latency(s):                           474.82
---------------Time to First Token----------------
Mean TTFT (ms):                          200415.03
Median TTFT (ms):                        235618.38
P95 TTFT (ms):                           296378.61
P99 TTFT (ms):                           299199.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          255.63
Median TPOT (ms):                        281.61
P95 TPOT (ms):                           353.91
P99 TPOT (ms):                           463.69
---------------Inter-token Latency----------------
Mean ITL (ms):                           59.97
Median ITL (ms):                         53.70
P95 ITL (ms):                            70.44
P99 ITL (ms):                            286.92
==================================================

normal

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
5
6
7
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--moe-a2a-backend=deepep
--max-running-requests=256
--deepep-mode=normal
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  2722.41
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              0.37
Output token throughput (tok/s):         378.79
Peak output token throughput (tok/s):    63639.56
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          3413.56
----------------------Latency---------------------
Mean Latency(s):                          1084.11
Median Latency(s):                        1206.16
P95 Latency(s):                           1510.02
P99 Latency(s):                           1897.83
---------------Time to First Token----------------
Mean TTFT (ms):                          773983.92
Median TTFT (ms):                        924140.14
P95 TTFT (ms):                           1214572.96
P99 TTFT (ms):                           1217017.99
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1058.70
Median TPOT (ms):                        1177.89
P95 TPOT (ms):                           1474.63
P99 TPOT (ms):                           1853.35
---------------Inter-token Latency----------------
Mean ITL (ms):                           303.15
Median ITL (ms):                         277.82
P95 ITL (ms):                            294.11
P99 ITL (ms):                            1229.38
==================================================
  • Итог: auto Общий TPS = 14451.05, normal Общий TPS = 3413.56. auto быстрее на 11037.49 tok/s (323.34%); Среднее TPOT = 255.63 ms против 1058.70 ms, снижение на 803.07 ms (75.85%).

Тип данных KV-кэша

bf16

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
5
6
    --tp-size=8
--dp-size=8
--enable-dp-attention
--moe-a2a-backend=deepep
--mem-fraction-static=0.85
--max-running-requests=256
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  657.33
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.52
Output token throughput (tok/s):         1603.58
Peak output token throughput (tok/s):    53623.54
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          14451.05
----------------------Latency---------------------
Mean Latency(s):                          261.76
Median Latency(s):                        288.37
P95 Latency(s):                           362.40
P99 Latency(s):                           474.82
---------------Time to First Token----------------
Mean TTFT (ms):                          200415.03
Median TTFT (ms):                        235618.38
P95 TTFT (ms):                           296378.61
P99 TTFT (ms):                           299199.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          255.63
Median TPOT (ms):                        281.61
P95 TPOT (ms):                           353.91
P99 TPOT (ms):                           463.69
---------------Inter-token Latency----------------
Mean ITL (ms):                           59.97
Median ITL (ms):                         53.70
P95 ITL (ms):                            70.44
P99 ITL (ms):                            286.92
==================================================

fp8_e4m3

  • Профиль: Balanced
  • Параметры бэкенда: 
    1
2
3
4
5
6
7
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--moe-a2a-backend=deepep
--max-running-requests=256
--kv-cache-dtype=fp8_e4m3
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  539.39
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.85
Output token throughput (tok/s):         1944.77
Peak output token throughput (tok/s):    60301.69
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          17525.76
----------------------Latency---------------------
Mean Latency(s):                          223.12
Median Latency(s):                        219.64
P95 Latency(s):                           327.25
P99 Latency(s):                           327.80
---------------Time to First Token----------------
Mean TTFT (ms):                          143076.43
Median TTFT (ms):                        151142.06
P95 TTFT (ms):                           234645.47
P99 TTFT (ms):                           239552.62
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          217.89
Median TPOT (ms):                        214.49
P95 TPOT (ms):                           319.58
P99 TPOT (ms):                           320.12
---------------Inter-token Latency----------------
Mean ITL (ms):                           78.24
Median ITL (ms):                         74.83
P95 ITL (ms):                            105.11
P99 ITL (ms):                            217.21
==================================================
  • Итог: fp8_e4m3 Общий TPS = 17525.76, bf16 Общий TPS = 14451.05. fp8_e4m3 быстрее на 3074.72 tok/s (21.28%); Среднее TPOT = 217.89 ms против 255.63 ms, снижение на 37.74 ms (14.76%).

Проверка точности KV-кэша FP8 (GSM8K)

На Hopper KV-кэш FP8 не используется по умолчанию (SGLang автоматически выбирает bf16); он повышает пропускную способность только при ограничении KV-кэшем — высокой конкурентности и длинном контексте. Чтобы проверить отсутствие ухудшения качества, мы провели контрольный тест GSM8K (200 примеров, режим размышлений выключен, жадное декодирование):

Тип данных KV Оценка GSM8K
bf16 0.975
fp8_e4m3 0.980

Разница находится в пределах шума (~1 вопрос / 200), то есть измеримого снижения точности не обнаружено. Тем не менее проверьте точность на своей нагрузке, особенно с длинным контекстом, где ошибка квантизации FP8 может накапливаться.

Сводка методов оптимизации

Сценарии бенчмарка После оптимизации Базовая конфигурация
Выбор движка инференса Общий TPS: 6487.53 (+18.34%)
Среднее TPOT(ms): 540.79		 Общий TPS: 5481.90
Среднее TPOT(ms): 636.24
Стратегия параллелизма Общий TPS: 7011.17 (+27.90%)
Среднее TPOT(ms): 497.48		 Общий TPS: 5481.90
Среднее TPOT(ms): 636.24
Доля памяти Общий TPS: 13760.98 (+151.03%)
Среднее TPOT(ms): 266.89		 Общий TPS: 5481.90
Среднее TPOT(ms): 636.24
Размер блока prefill Общий TPS: 13760.98 (+151.03%)
Среднее TPOT(ms): 266.89		 Общий TPS: 5481.90
Среднее TPOT(ms): 636.24
DeepEP и максимум одновременных запросов Общий TPS: 14451.05 (+163.61%)
Среднее TPOT(ms): 255.63		 Общий TPS: 5481.90
Среднее TPOT(ms): 636.24
Режим DeepEP Общий TPS: 14451.05 (+163.61%)
Среднее TPOT(ms): 255.63		 Общий TPS: 5481.90
Среднее TPOT(ms): 636.24
Тип данных KV-кэша Общий TPS: 17525.76 (+219.70%)
Среднее TPOT(ms): 217.89		 Общий TPS: 5481.90
Среднее TPOT(ms): 636.24

Другие сценарии бенчмарка

Balanced: 8K входных / 1K выходных (цель оптимизации)

  • Базовые параметры бэкенда: 
    1
    --tensor-parallel-size=8
Результат базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  1684.76
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              0.59
Output token throughput (tok/s):         608.31
Peak output token throughput (tok/s):    21900.04
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          5481.90
----------------------Latency---------------------
Mean Latency(s):                          651.51
Median Latency(s):                        823.57
P95 Latency(s):                           857.92
P99 Latency(s):                           858.57
---------------Time to First Token----------------
Mean TTFT (ms):                          592315.54
Median TTFT (ms):                        762631.30
P95 TTFT (ms):                           799738.92
P99 TTFT (ms):                           800277.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          636.24
Median TPOT (ms):                        804.27
P95 TPOT (ms):                           837.81
P99 TPOT (ms):                           838.45
---------------Inter-token Latency----------------
Mean ITL (ms):                           57.86
Median ITL (ms):                         56.85
P95 ITL (ms):                            69.41
P99 ITL (ms):                            90.01
==================================================
  • Параметры бэкенда после оптимизации: 
    1
2
3
4
5
6
7
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--moe-a2a-backend=deepep
--max-running-requests=256
--kv-cache-dtype=fp8_e4m3
Результат бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  539.39
Total input tokens:                      8204000
Total generated tokens:                  1024000
Request throughput (req/s):              1.85
Output token throughput (tok/s):         1944.77
Peak output token throughput (tok/s):    60301.69
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          17525.76
----------------------Latency---------------------
Mean Latency(s):                          223.12
Median Latency(s):                        219.64
P95 Latency(s):                           327.25
P99 Latency(s):                           327.80
---------------Time to First Token----------------
Mean TTFT (ms):                          143076.43
Median TTFT (ms):                        151142.06
P95 TTFT (ms):                           234645.47
P99 TTFT (ms):                           239552.62
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          217.89
Median TPOT (ms):                        214.49
P95 TPOT (ms):                           319.58
P99 TPOT (ms):                           320.12
---------------Inter-token Latency----------------
Mean ITL (ms):                           78.24
Median ITL (ms):                         74.83
P95 ITL (ms):                            105.11
P99 ITL (ms):                            217.21
==================================================

Короткий контекст (1K входных / 128 выходных)

  • Базовые параметры бэкенда:N/A
Результат базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  88.45
Total input tokens:                      1036000
Total generated tokens:                  128000
Request throughput (req/s):              11.29
Output token throughput (tok/s):         1451.20
Peak output token throughput (tok/s):    102180.57
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          13196.90
----------------------Latency---------------------
Mean Latency(s):                          40.54
Median Latency(s):                        39.42
P95 Latency(s):                           57.97
P99 Latency(s):                           60.17
---------------Time to First Token----------------
Mean TTFT (ms):                          19596.45
Median TTFT (ms):                        22675.56
P95 TTFT (ms):                           34452.65
P99 TTFT (ms):                           36690.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          316.73
Median TPOT (ms):                        307.96
P95 TPOT (ms):                           452.86
P99 TPOT (ms):                           470.04
---------------Inter-token Latency----------------
Mean ITL (ms):                           164.92
Median ITL (ms):                         179.87
P95 ITL (ms):                            185.20
P99 ITL (ms):                            223.80
==================================================
  • Параметры бэкенда после оптимизации: 
    1
2
3
4
5
6
7
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--moe-a2a-backend=deepep
--max-running-requests=256
--kv-cache-dtype=fp8_e4m3
Результат бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  65.34
Total input tokens:                      1036000
Total generated tokens:                  128000
Request throughput (req/s):              15.29
Output token throughput (tok/s):         2009.80
Peak output token throughput (tok/s):    54517.94
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          18276.60
----------------------Latency---------------------
Mean Latency(s):                          27.02
Median Latency(s):                        29.93
P95 Latency(s):                           30.87
P99 Latency(s):                           44.75
---------------Time to First Token----------------
Mean TTFT (ms):                          17437.67
Median TTFT (ms):                        19408.58
P95 TTFT (ms):                           24509.18
P99 TTFT (ms):                           31027.97
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          211.06
Median TPOT (ms):                        233.85
P95 TPOT (ms):                           241.16
P99 TPOT (ms):                           349.60
---------------Inter-token Latency----------------
Mean ITL (ms):                           75.41
Median ITL (ms):                         75.81
P95 ITL (ms):                            109.63
P99 ITL (ms):                            113.08
==================================================

Длинный контекст (32K входных / 100 выходных)

  • Базовые параметры бэкенда:N/A
Результат базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             77
Benchmark duration (s):                  308.44
Total input tokens:                      3201200
Total generated tokens:                  10000
Request throughput (req/s):              0.32
Output token throughput (tok/s):         32.74
Peak output token throughput (tok/s):    4245.77
Peak concurrent requests:                77.00
Total Token throughput (tok/s):          10513.68
----------------------Latency---------------------
Mean Latency(s):                          132.00
Median Latency(s):                        131.48
P95 Latency(s):                           214.57
P99 Latency(s):                           217.83
---------------Time to First Token----------------
Mean TTFT (ms):                          106054.23
Median TTFT (ms):                        104360.05
P95 TTFT (ms):                           198608.95
P99 TTFT (ms):                           206310.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1320.02
Median TPOT (ms):                        1314.76
P95 TPOT (ms):                           2145.71
P99 TPOT (ms):                           2178.26
---------------Inter-token Latency----------------
Mean ITL (ms):                           262.10
Median ITL (ms):                         274.05
P95 ITL (ms):                            281.04
P99 ITL (ms):                            281.32
==================================================
  • Параметры бэкенда после оптимизации: 
    1
2
3
4
5
6
7
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--moe-a2a-backend=deepep
--max-running-requests=256
--kv-cache-dtype=fp8_e4m3
Результат бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             77
Benchmark duration (s):                  135.63
Total input tokens:                      3201200
Total generated tokens:                  10000
Request throughput (req/s):              0.74
Output token throughput (tok/s):         82.32
Peak output token throughput (tok/s):    13906.10
Peak concurrent requests:                77.00
Total Token throughput (tok/s):          26433.15
----------------------Latency---------------------
Mean Latency(s):                          53.73
Median Latency(s):                        53.63
P95 Latency(s):                           76.27
P99 Latency(s):                           78.63
---------------Time to First Token----------------
Mean TTFT (ms):                          23225.41
Median TTFT (ms):                        20088.83
P95 TTFT (ms):                           34063.13
P99 TTFT (ms):                           35622.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          537.26
Median TPOT (ms):                        536.30
P95 TPOT (ms):                           762.65
P99 TPOT (ms):                           786.26
---------------Inter-token Latency----------------
Mean ITL (ms):                           308.09
Median ITL (ms):                         304.06
P95 ITL (ms):                            621.19
P99 ITL (ms):                            625.53
==================================================

Интенсивная генерация (1K входных / 2K выходных)

  • Базовые параметры бэкенда:N/A
Результат базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             91
Benchmark duration (s):                  265.81
Total input tokens:                      202400
Total generated tokens:                  400000
Request throughput (req/s):              0.75
Output token throughput (tok/s):         1506.10
Peak output token throughput (tok/s):    41424.97
Peak concurrent requests:                91.00
Total Token throughput (tok/s):          2268.18
----------------------Latency---------------------
Mean Latency(s):                          84.22
Median Latency(s):                        86.45
P95 Latency(s):                           90.73
P99 Latency(s):                           90.83
---------------Time to First Token----------------
Mean TTFT (ms):                          236.63
Median TTFT (ms):                        235.39
P95 TTFT (ms):                           254.39
P99 TTFT (ms):                           257.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          42.11
Median TPOT (ms):                        43.22
P95 TPOT (ms):                           45.37
P99 TPOT (ms):                           45.42
---------------Inter-token Latency----------------
Mean ITL (ms):                           42.01
Median ITL (ms):                         43.13
P95 ITL (ms):                            45.27
P99 ITL (ms):                            45.32
==================================================
  • Параметры бэкенда после оптимизации: 
    1
2
3
4
5
6
7
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--moe-a2a-backend=deepep
--max-running-requests=256
--kv-cache-dtype=fp8_e4m3
Результат бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             103
Benchmark duration (s):                  266.01
Total input tokens:                      202400
Total generated tokens:                  400000
Request throughput (req/s):              0.75
Output token throughput (tok/s):         1505.53
Peak output token throughput (tok/s):    33577.32
Peak concurrent requests:                103.00
Total Token throughput (tok/s):          2267.33
----------------------Latency---------------------
Mean Latency(s):                          93.09
Median Latency(s):                        96.15
P95 Latency(s):                           102.79
P99 Latency(s):                           102.95
---------------Time to First Token----------------
Mean TTFT (ms):                          374.11
Median TTFT (ms):                        355.12
P95 TTFT (ms):                           479.63
P99 TTFT (ms):                           747.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          46.54
Median TPOT (ms):                        48.07
P95 TPOT (ms):                           51.40
P99 TPOT (ms):                           51.47
---------------Inter-token Latency----------------
Mean ITL (ms):                           46.38
Median ITL (ms):                         47.92
P95 ITL (ms):                            51.24
P99 ITL (ms):                            51.32
==================================================

ShareGPT

  • Базовые параметры бэкенда:N/A
Результат базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  99.70
Total input tokens:                      328145
Total generated tokens:                  270248
Request throughput (req/s):              10.03
Output token throughput (tok/s):         2728.23
Peak output token throughput (tok/s):    99078.05
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          6040.94
----------------------Latency---------------------
Mean Latency(s):                          40.33
Median Latency(s):                        36.92
P95 Latency(s):                           80.82
P99 Latency(s):                           89.73
---------------Time to First Token----------------
Mean TTFT (ms):                          3745.78
Median TTFT (ms):                        1735.57
P95 TTFT (ms):                           11581.05
P99 TTFT (ms):                           13380.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          149.24
Median TPOT (ms):                        140.13
P95 TPOT (ms):                           242.35
P99 TPOT (ms):                           318.37
---------------Inter-token Latency----------------
Mean ITL (ms):                           135.88
Median ITL (ms):                         132.05
P95 ITL (ms):                            209.02
P99 ITL (ms):                            246.28
==================================================
  • Параметры бэкенда после оптимизации: 
    1
2
3
4
5
6
7
    --tp-size=8
--dp-size=8
--enable-dp-attention
--mem-fraction-static=0.85
--moe-a2a-backend=deepep
--max-running-requests=256
--kv-cache-dtype=fp8_e4m3
Результат бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  211.07
Total input tokens:                      328145
Total generated tokens:                  270248
Request throughput (req/s):              4.74
Output token throughput (tok/s):         1285.09
Peak output token throughput (tok/s):    33224.09
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          2845.48
----------------------Latency---------------------
Mean Latency(s):                          89.51
Median Latency(s):                        87.50
P95 Latency(s):                           162.23
P99 Latency(s):                           186.86
---------------Time to First Token----------------
Mean TTFT (ms):                          41335.55
Median TTFT (ms):                        53587.71
P95 TTFT (ms):                           71490.14
P99 TTFT (ms):                           79840.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          331.21
Median TPOT (ms):                        270.24
P95 TPOT (ms):                           620.79
P99 TPOT (ms):                           1143.93
---------------Inter-token Latency----------------
Mean ITL (ms):                           178.92
Median ITL (ms):                         205.22
P95 ITL (ms):                            223.71
P99 ITL (ms):                            239.28
==================================================