Оптимизация пропускной способности Qwen3.5-9B

Вывод

Рекомендуемая конфигурация для оптимизации пропускной способности Qwen/Qwen3.5-9B на NVIDIA H100 80GB HBM3:

Команда обслуживания модели

vllm serve Qwen/Qwen3.5-9B \
    --reasoning-parser=qwen3 \
    --performance-mode=throughput \
    --max-model-len=32768 \
    --max-num-seqs=512

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка	базовая конфигурация (vLLM без оптимизации)	После оптимизации
Профиль ShareGPT	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12	Общий TPS: 11768.22 (+2.09%) Среднее TPOT(ms): 15.85
Профиль высокой пропускной способности	Общий TPS: 29822.16 Среднее TPOT(ms): 34.63	Общий TPS: 34470.14 (+15.59%) Среднее TPOT(ms): 36.97
Профиль длинного контекста	Общий TPS: 32416.84 Среднее TPOT(ms): 10.21	Общий TPS: 32658.73 (+0.75%) Среднее TPOT(ms): 9.97
Профиль интенсивной генерации	Общий TPS: 2973.42 Среднее TPOT(ms): 0.33	Общий TPS: 3025.11 (+1.74%) Среднее TPOT(ms): 0.12

Примечание

Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.
Квантизация может влиять на точность, однако для большинства моделей FP8 обеспечивает снижение точности менее чем на 1%. Подробности см. в результатах оценки. Поэтому для обслуживания с низкой задержкой настоятельно рекомендуется квантизация FP8.
Спекулятивное декодирование может существенно уменьшить задержку при низкой конкурентности запросов. Однако ускорение зависит от распределения данных в бенчмарке и выбора черновой модели. Например, выбранная здесь черновая модель обучена на английских данных, поэтому для других языков результат может быть хуже оптимального.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Условия эксперимента

Модель

Qwen/Qwen3.5-9B

Оборудование

NVIDIA H100 80GB HBM3

Версия движка

vLLM v0.17.1
SGLang v0.5.9

Методика бенчмарка

В этом проекте используется функция запуска бенчмарка GPUStack одним щелчком для нагрузок обслуживания моделей. Бенчмарки этого документа выполнялись именно таким способом.

Бенчмарки GPUStack реализованы на основе guidellm через проект-обёртку benchmark-runner.

Для перечисленных ниже конфигураций GPUStack выполняет развёртывание модели, отправку заданий бенчмарка и сбор результатов.

Профили бенчмарка

ShareGPT

dataset_name: ShareGPT
request_rate: 1000
total_requests: 1000

Пропускная способность

dataset_name: Random
dataset_input_tokens: 1024
dataset_output_tokens: 128
dataset_seed: 42
request_rate: 1000
total_requests: 1000

Длинный контекст

dataset_name: Random
dataset_input_tokens: 32000
dataset_output_tokens: 100
dataset_seed: 42
request_rate: 1
total_requests: 100

Интенсивная генерация

dataset_name: Random
dataset_input_tokens: 1000
dataset_output_tokens: 2000
dataset_seed: 42
request_rate: 1
total_requests: 200

Результаты эксперимента

Выбор движка инференса

vLLM

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.74
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.27
Output token throughput (tok/s):         5202.94
Peak output token throughput (tok/s):    20132659.20
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11527.14
----------------------Latency---------------------
Mean Latency(s):                          24.11
Median Latency(s):                        21.90
P99 Latency(s):                           51.16
---------------Time to First Token----------------
Mean TTFT (ms):                          1604.23
Median TTFT (ms):                        646.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5737.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.12
Median TPOT (ms):                        3.42
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.68
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.45
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            118.73
==================================================

SGLang

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--context-length=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  96.01
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              10.42
Output token throughput (tok/s):         2933.60
Peak output token throughput (tok/s):    92831.68
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          6499.41
----------------------Latency---------------------
Mean Latency(s):                          39.08
Median Latency(s):                        41.18
P99 Latency(s):                           74.40
---------------Time to First Token----------------
Mean TTFT (ms):                          26217.15
Median TTFT (ms):                        34401.34
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           41101.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          138.86
Median TPOT (ms):                        112.19
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           591.70
---------------Inter-token Latency----------------
Mean ITL (ms):                           45.86
Median ITL (ms):                         49.31
P95 ITL (ms):                            N/A
P99 ITL (ms):                            59.71
==================================================

Итог: vLLM Общий TPS = 11527.14, SGLang Общий TPS = 6499.41. vLLM быстрее на 5027.72 tok/s (77.36%); Среднее TPOT = 15.12 ms против 138.86 ms, снижение на 123.74 ms (89.11%).

Кэш префиксов

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--enable-prefix-caching
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  56.47
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              17.71
Output token throughput (tok/s):         5066.42
Peak output token throughput (tok/s):    31242972.50
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11224.70
----------------------Latency---------------------
Mean Latency(s):                          24.90
Median Latency(s):                        22.40
P99 Latency(s):                           52.82
---------------Time to First Token----------------
Mean TTFT (ms):                          1995.45
Median TTFT (ms):                        1341.75
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           6549.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.66
Median TPOT (ms):                        5.21
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           130.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.60
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            120.90
==================================================

Спекулятивное декодирование

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--speculative-config={"method":"mtp","num_speculative_tokens":4}
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     514
Maximum request concurrency:             512
Benchmark duration (s):                  12.27
Total input tokens:                      23313
Total generated tokens:                  131530
Request throughput (req/s):              41.90
Output token throughput (tok/s):         12792.49
Peak output token throughput (tok/s):    13918875.50
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          15059.89
----------------------Latency---------------------
Mean Latency(s):                          11.77
Median Latency(s):                        11.81
P99 Latency(s):                           12.07
---------------Time to First Token----------------
Mean TTFT (ms):                          268.95
Median TTFT (ms):                        0.00
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           3010.96
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          42.50
Median TPOT (ms):                        31.35
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           214.99
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Режим производительности

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.66
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.29
Output token throughput (tok/s):         5189.06
Peak output token throughput (tok/s):    30482939.47
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11496.39
----------------------Latency---------------------
Mean Latency(s):                          24.00
Median Latency(s):                        21.51
P95 Latency(s):                           N/A
P99 Latency(s):                           51.01
---------------Time to First Token----------------
Mean TTFT (ms):                          1824.24
Median TTFT (ms):                        982.10
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5550.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.41
Median TPOT (ms):                        4.53
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.10
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.96
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            115.72
==================================================

Максимальное число последовательностей

Последовательности 512

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-seqs=512

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.89
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.22
Output token throughput (tok/s):         5174.68
Peak output token throughput (tok/s):    21197768.76
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11464.53
----------------------Latency---------------------
Mean Latency(s):                          24.11
Median Latency(s):                        21.96
P95 Latency(s):                           48.41
P99 Latency(s):                           51.17
---------------Time to First Token----------------
Mean TTFT (ms):                          1893.47
Median TTFT (ms):                        1233.85
P95 TTFT (ms):                           5659.91
P99 TTFT (ms):                           6222.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.18
Median TPOT (ms):                        4.48
P95 TPOT (ms):                           91.71
P99 TPOT (ms):                           127.60
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.48
Median ITL (ms):                         0.00
P95 ITL (ms):                            83.92
P99 ITL (ms):                            116.65
==================================================

Последовательности 1024

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-seqs=1024

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.87
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.23
Output token throughput (tok/s):         5205.42
Peak output token throughput (tok/s):    22885338.88
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11532.64
----------------------Latency---------------------
Mean Latency(s):                          24.15
Median Latency(s):                        21.71
P95 Latency(s):                           48.49
P99 Latency(s):                           51.24
---------------Time to First Token----------------
Mean TTFT (ms):                          1864.39
Median TTFT (ms):                        1137.51
P95 TTFT (ms):                           5803.37
P99 TTFT (ms):                           6125.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.87
Median TPOT (ms):                        4.70
P95 TPOT (ms):                           93.65
P99 TPOT (ms):                           127.45
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.28
Median ITL (ms):                         0.00
P95 ITL (ms):                            87.39
P99 ITL (ms):                            116.13
==================================================

Последовательности 2048

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-seqs=2048

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  55.28
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.09
Output token throughput (tok/s):         5140.19
Peak output token throughput (tok/s):    23553347.13
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11388.13
----------------------Latency---------------------
Mean Latency(s):                          24.34
Median Latency(s):                        22.16
P95 Latency(s):                           48.86
P99 Latency(s):                           51.63
---------------Time to First Token----------------
Mean TTFT (ms):                          1853.85
Median TTFT (ms):                        1166.25
P95 TTFT (ms):                           5543.84
P99 TTFT (ms):                           6126.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.91
Median TPOT (ms):                        4.35
P95 TPOT (ms):                           100.86
P99 TPOT (ms):                           129.26
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.36
Median ITL (ms):                         0.00
P95 ITL (ms):                            94.30
P99 ITL (ms):                            118.29
==================================================

Итог: Seqs 1024 Общий TPS = 11532.64, Seqs 2048 Общий TPS = 11388.13. Seqs 1024 быстрее на 144.51 tok/s (1.27%); Среднее TPOT = 16.87 ms против 16.91 ms, снижение на 0.04 ms (0.25%).

Максимум токенов в батче и Режим производительности

24k

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=24576

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.56
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.33
Output token throughput (tok/s):         5196.51
Peak output token throughput (tok/s):    32356059.43
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11512.90
----------------------Latency---------------------
Mean Latency(s):                          23.98
Median Latency(s):                        21.78
P99 Latency(s):                           50.95
---------------Time to First Token----------------
Mean TTFT (ms):                          1870.77
Median TTFT (ms):                        1771.31
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5297.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.83
Median TPOT (ms):                        4.86
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           125.06
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.22
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            113.92
==================================================

32k

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.20
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.45
Output token throughput (tok/s):         5268.65
Peak output token throughput (tok/s):    29985425.61
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11672.72
----------------------Latency---------------------
Mean Latency(s):                          23.72
Median Latency(s):                        21.21
P99 Latency(s):                           50.44
---------------Time to First Token----------------
Mean TTFT (ms):                          1970.35
Median TTFT (ms):                        1344.71
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5206.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.53
Median TPOT (ms):                        5.04
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           125.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.56
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            111.59
==================================================

Итог: 32k Общий TPS = 11672.72, 24k Общий TPS = 11512.90. 32k быстрее на 159.82 tok/s (1.39%); Среднее TPOT = 16.53 ms против 15.83 ms, увеличение на 0.70 ms (4.43% медленнее).

Максимальное число последовательностей и Режим производительности

Максимальное число последовательностей 256

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=256

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  57.32
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              17.45
Output token throughput (tok/s):         4973.66
Peak output token throughput (tok/s):    30062067.31
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11019.18
----------------------Latency---------------------
Mean Latency(s):                          23.57
Median Latency(s):                        22.86
P99 Latency(s):                           49.24
---------------Time to First Token----------------
Mean TTFT (ms):                          11061.67
Median TTFT (ms):                        14983.14
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           18807.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          44.58
Median TPOT (ms):                        31.66
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           229.02
---------------Inter-token Latency----------------
Mean ITL (ms):                           5.29
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            54.12
==================================================

Максимальное число последовательностей 512

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  53.72
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.62
Output token throughput (tok/s):         5311.75
Peak output token throughput (tok/s):    23159852.52
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11768.22
----------------------Latency---------------------
Mean Latency(s):                          23.50
Median Latency(s):                        21.13
P99 Latency(s):                           50.04
---------------Time to First Token----------------
Mean TTFT (ms):                          1826.53
Median TTFT (ms):                        1004.67
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5582.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.85
Median TPOT (ms):                        4.49
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           123.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.39
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            112.55
==================================================

Итог: Max Num Seqs 512 Общий TPS = 11768.22, Max Num Seqs 256 Общий TPS = 11019.18. Max Num Seqs 512 быстрее на 749.05 tok/s (6.80%); Среднее TPOT = 15.85 ms против 44.58 ms, снижение на 28.73 ms (64.45%).

Максимальное число последовательностей и Максимум токенов в батче

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=32768
--max-num-seqs=512

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  55.33
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.07
Output token throughput (tok/s):         5115.71
Peak output token throughput (tok/s):    34751436.57
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11333.89
----------------------Latency---------------------
Mean Latency(s):                          24.34
Median Latency(s):                        22.08
P99 Latency(s):                           51.69
---------------Time to First Token----------------
Mean TTFT (ms):                          1831.81
Median TTFT (ms):                        1345.90
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5161.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.39
Median TPOT (ms):                        4.48
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.14
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.92
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            116.48
==================================================

Максимальное число последовательностей и Максимум токенов в батче и Режим производительности

Токенов в батче 32k, последовательностей 512

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=32768
--max-num-seqs=512

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  55.33
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.07
Output token throughput (tok/s):         5115.71
Peak output token throughput (tok/s):    34751436.57
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11333.89
----------------------Latency---------------------
Mean Latency(s):                          24.34
Median Latency(s):                        22.08
P95 Latency(s):                           N/A
P99 Latency(s):                           51.69
---------------Time to First Token----------------
Mean TTFT (ms):                          1831.81
Median TTFT (ms):                        1345.90
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5161.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.39
Median TPOT (ms):                        4.48
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.14
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.92
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            116.48
==================================================

Токенов в батче 48k, последовательностей 512

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=49152
--max-num-seqs=512

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.48
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.36
Output token throughput (tok/s):         5223.82
Peak output token throughput (tok/s):    33459752.45
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11573.41
----------------------Latency---------------------
Mean Latency(s):                          23.96
Median Latency(s):                        21.63
P95 Latency(s):                           N/A
P99 Latency(s):                           50.92
---------------Time to First Token----------------
Mean TTFT (ms):                          2022.91
Median TTFT (ms):                        2480.42
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5146.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.20
Median TPOT (ms):                        5.63
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           124.01
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.04
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            111.00
==================================================

Токенов в батче 48k, последовательностей 768

Профиль: ShareGPT

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=49152
--max-num-seqs=768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.07
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.49
Output token throughput (tok/s):         5248.67
Peak output token throughput (tok/s):    36318641.50
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11628.47
----------------------Latency---------------------
Mean Latency(s):                          23.72
Median Latency(s):                        21.75
P95 Latency(s):                           N/A
P99 Latency(s):                           50.48
---------------Time to First Token----------------
Mean TTFT (ms):                          1817.87
Median TTFT (ms):                        1821.59
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5041.22
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.65
Median TPOT (ms):                        4.57
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           123.59
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.23
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            114.09
==================================================

Итог: Batch Token 48k and Seqs 768 Общий TPS = 11628.47, Batch Token 32k and Seqs 512 Общий TPS = 11333.89. Batch Token 48k and Seqs 768 быстрее на 294.57 tok/s (2.60%); Среднее TPOT = 15.65 ms против 16.39 ms, снижение на 0.74 ms (4.51%).

Сводка методов оптимизации

Сценарии бенчмарка	После оптимизации	Базовая конфигурация
Выбор движка инференса	Общий TPS: 11527.14 (+0.00%) Среднее TPOT(ms): 15.12	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12
Кэш префиксов	Общий TPS: 11224.70 (-2.62%) Среднее TPOT(ms): 16.66	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12
Спекулятивное декодирование	Общий TPS: 15059.89 (Доля успешных запросов: 51.5%, оптимизация пропущена) Среднее TPOT(ms): 42.50	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12
Режим производительности	Общий TPS: 11496.39 (-0.27%) Среднее TPOT(ms): 16.41	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12
Максимальное число последовательностей	Общий TPS: 11532.64 (+0.05%) Среднее TPOT(ms): 16.87	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12
Максимум токенов в батче && Режим производительности	Общий TPS: 11672.72 (+1.26%) Среднее TPOT(ms): 16.53	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12
Максимальное число последовательностей && Режим производительности	Общий TPS: 11768.22 (+2.09%) Среднее TPOT(ms): 15.85	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12
Максимальное число последовательностей && Максимум токенов в батче	Общий TPS: 11333.89 (-1.68%) Среднее TPOT(ms): 16.39	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12
Максимальное число последовательностей && Максимум токенов в батче && Режим производительности	Общий TPS: 11628.47 (+0.88%) Среднее TPOT(ms): 15.65	Общий TPS: 11527.14 Среднее TPOT(ms): 15.12

Другие сценарии бенчмарка

Профиль ShareGPT

Базовые параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат базового бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.74
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.27
Output token throughput (tok/s):         5202.94
Peak output token throughput (tok/s):    20132659.20
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11527.14
----------------------Latency---------------------
Mean Latency(s):                          24.11
Median Latency(s):                        21.90
P99 Latency(s):                           51.16
---------------Time to First Token----------------
Mean TTFT (ms):                          1604.23
Median TTFT (ms):                        646.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5737.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.12
Median TPOT (ms):                        3.42
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.68
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.45
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            118.73
==================================================

Параметры бэкенда после оптимизации:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Результат бенчмарка после оптимизации

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  53.72
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.62
Output token throughput (tok/s):         5311.75
Peak output token throughput (tok/s):    23159852.52
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11768.22
----------------------Latency---------------------
Mean Latency(s):                          23.50
Median Latency(s):                        21.13
P99 Latency(s):                           50.04
---------------Time to First Token----------------
Mean TTFT (ms):                          1826.53
Median TTFT (ms):                        1004.67
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5582.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.85
Median TPOT (ms):                        4.49
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           123.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.39
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            112.55
==================================================

Профиль высокой пропускной способности

Базовые параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат базового бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  43.00
Total input tokens:                      1033932
Total generated tokens:                  128000
Request throughput (req/s):              23.26
Output token throughput (tok/s):         3285.25
Peak output token throughput (tok/s):    4418690.63
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          29822.16
----------------------Latency---------------------
Mean Latency(s):                          21.27
Median Latency(s):                        21.11
P99 Latency(s):                           36.40
---------------Time to First Token----------------
Mean TTFT (ms):                          4433.02
Median TTFT (ms):                        1510.44
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           14914.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          34.63
Median TPOT (ms):                        11.80
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           116.52
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Параметры бэкенда после оптимизации:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Результат бенчмарка после оптимизации

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  39.10
Total input tokens:                      1033932
Total generated tokens:                  128000
Request throughput (req/s):              25.55
Output token throughput (tok/s):         3797.28
Peak output token throughput (tok/s):    7724761.32
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          34470.14
----------------------Latency---------------------
Mean Latency(s):                          19.38
Median Latency(s):                        19.24
P99 Latency(s):                           31.36
---------------Time to First Token----------------
Mean TTFT (ms):                          4731.56
Median TTFT (ms):                        2442.61
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           13309.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          36.97
Median TPOT (ms):                        19.08
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           103.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Профиль длинного контекста

Базовые параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат базового бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             9
Benchmark duration (s):                  100.89
Total input tokens:                      3200993
Total generated tokens:                  10000
Request throughput (req/s):              0.99
Output token throughput (tok/s):         100.96
Peak output token throughput (tok/s):    198.08
Peak concurrent requests:                9.00
Total Token throughput (tok/s):          32416.84
----------------------Latency---------------------
Mean Latency(s):                          7.88
Median Latency(s):                        8.10
P95 Latency(s):                           N/A
P99 Latency(s):                           8.54
---------------Time to First Token----------------
Mean TTFT (ms):                          1021.12
Median TTFT (ms):                        1020.12
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           1041.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.21
Median TPOT (ms):                        10.20
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           10.42
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Параметры бэкенда после оптимизации:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Результат бенчмарка после оптимизации

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             14
Benchmark duration (s):                  100.13
Total input tokens:                      3200993
Total generated tokens:                  10000
Request throughput (req/s):              1.00
Output token throughput (tok/s):         101.71
Peak output token throughput (tok/s):    227.24
Peak concurrent requests:                14.00
Total Token throughput (tok/s):          32658.73
----------------------Latency---------------------
Mean Latency(s):                          8.00
Median Latency(s):                        8.00
P99 Latency(s):                           12.36
---------------Time to First Token----------------
Mean TTFT (ms):                          997.50
Median TTFT (ms):                        981.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           1421.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.97
Median TPOT (ms):                        9.81
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           14.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Профиль интенсивной генерации

Базовые параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат базового бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             18
Benchmark duration (s):                  214.96
Total input tokens:                      201982
Total generated tokens:                  400000
Request throughput (req/s):              0.93
Output token throughput (tok/s):         1975.75
Peak output token throughput (tok/s):    123795.17
Peak concurrent requests:                18.00
Total Token throughput (tok/s):          2973.42
----------------------Latency---------------------
Mean Latency(s):                          17.48
Median Latency(s):                        17.59
P99 Latency(s):                           17.60
---------------Time to First Token----------------
Mean TTFT (ms):                          54.95
Median TTFT (ms):                        55.17
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           59.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.33
Median TPOT (ms):                        0.03
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           8.79
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.31
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            8.77
==================================================

Параметры бэкенда после оптимизации:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Результат бенчмарка после оптимизации

============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             18
Benchmark duration (s):                  214.98
Total input tokens:                      201982
Total generated tokens:                  400000
Request throughput (req/s):              0.93
Output token throughput (tok/s):         2010.10
Peak output token throughput (tok/s):    59750.47
Peak concurrent requests:                18.00
Total Token throughput (tok/s):          3025.11
----------------------Latency---------------------
Mean Latency(s):                          17.55
Median Latency(s):                        17.66
P99 Latency(s):                           17.67
---------------Time to First Token----------------
Mean TTFT (ms):                          55.85
Median TTFT (ms):                        56.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           60.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.12
Median TPOT (ms):                        0.03
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           0.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.09
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================