Оптимизация задержки Qwen3.5-35B-A3B

Вывод

Рекомендуемая конфигурация для оптимизации задержки Qwen/Qwen3.5-35B-A3B-FP8 на NVIDIA H200:

Команда обслуживания модели

vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \
    --reasoning-parser=qwen3 \
    --speculative-config={"method":"mtp","num_speculative_tokens":1} \
    --max-model-len=32768

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка	Базовая конфигурация (vLLM без оптимизации)	После оптимизации
Rate 1	Средняя задержка: 1.85s/req	Средняя задержка: 1.50s/req (1.23x faster)
Rate 4	Средняя задержка: 3.30s/req	Средняя задержка: 2.52s/req (1.31x faster)
Rate 8	Средняя задержка: 5.32s/req	Средняя задержка: 3.28s/req (1.63x faster)
Rate 16	Средняя задержка: 18.59s/req	Средняя задержка: 6.79s/req (2.74x faster)

Примечание

Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.
Квантизация может влиять на точность, однако для большинства моделей FP8 обеспечивает снижение точности менее чем на 1%. Подробности см. в результатах оценки. Поэтому для обслуживания с низкой задержкой настоятельно рекомендуется квантизация FP8.
Спекулятивное декодирование может существенно уменьшить задержку при низкой конкурентности запросов. Однако ускорение зависит от распределения данных в бенчмарке и выбора черновой модели. Например, выбранная здесь черновая модель обучена на английских данных, поэтому для других языков результат может быть хуже оптимального.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Условия эксперимента

Модель

Qwen/Qwen3.5-35B-A3B

Оборудование

NVIDIA H200

Версия движка

vLLM v0.17.1
SGLang v0.5.9

Методика бенчмарка

В этом проекте используется функция запуска бенчмарка GPUStack одним щелчком для нагрузок обслуживания моделей. Бенчмарки этого документа выполнялись именно таким способом.

Бенчмарки GPUStack реализованы на основе guidellm через проект-обёртку benchmark-runner.

Для перечисленных ниже конфигураций GPUStack выполняет развёртывание модели, отправку заданий бенчмарка и сбор результатов.

Профили бенчмарка

ShareGPT 8RPS

dataset_name: ShareGPT
request_rate: 8
total_requests: 1000

ShareGPT 1RPS

dataset_name: ShareGPT
request_rate: 1
total_requests: 1000

ShareGPT 4RPS

dataset_name: ShareGPT
request_rate: 4
total_requests: 1000

ShareGPT 16RPS

dataset_name: ShareGPT
request_rate: 16
total_requests: 1000

Замена открытыми инструментами

Если вы не используете GPUStack, его процесс запуска бенчмарков можно заменить прямыми командами guidellm benchmark.

Для профилей с dataset_name: ShareGPT:

guidellm benchmark \
  --target ${target} \
  --profile constant \
  --rate ${request_rate} \
  --max-requests ${total_requests} \
  --processor ${model_path} \
  --data ./ShareGPT_V3_unfiltered_cleaned_split.json

Результаты эксперимента

Выбор движка инференса

vLLM

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             59
Benchmark duration (s):                  132.25
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.56
Output token throughput (tok/s):         2137.15
Peak output token throughput (tok/s):    1410355.61
Peak concurrent requests:                59.00
Total Token throughput (tok/s):          4734.87
----------------------Latency---------------------
Mean Latency(s):                          5.32
Median Latency(s):                        4.72
P95 Latency(s):                           13.39
P99 Latency(s):                           16.27
---------------Time to First Token----------------
Mean TTFT (ms):                          68.48
Median TTFT (ms):                        62.59
P95 TTFT (ms):                           102.52
P99 TTFT (ms):                           131.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.35
Median TPOT (ms):                        0.20
P95 TPOT (ms):                           18.88
P99 TPOT (ms):                           20.04
---------------Inter-token Latency----------------
Mean ITL (ms):                           2.11
Median ITL (ms):                         0.00
P95 ITL (ms):                            18.78
P99 ITL (ms):                            19.94
==================================================

SGLang

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--context-length=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             246
Benchmark duration (s):                  148.88
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              6.72
Output token throughput (tok/s):         1896.19
Peak output token throughput (tok/s):    36196.65
Peak concurrent requests:                246.00
Total Token throughput (tok/s):          4201.01
----------------------Latency---------------------
Mean Latency(s):                          25.15
Median Latency(s):                        23.44
P95 Latency(s):                           50.27
P99 Latency(s):                           61.67
---------------Time to First Token----------------
Mean TTFT (ms):                          6456.42
Median TTFT (ms):                        6537.59
P95 TTFT (ms):                           12387.08
P99 TTFT (ms):                           12670.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          89.37
Median TPOT (ms):                        82.58
P95 TPOT (ms):                           134.49
P99 TPOT (ms):                           222.06
---------------Inter-token Latency----------------
Mean ITL (ms):                           66.66
Median ITL (ms):                         68.70
P95 ITL (ms):                            93.84
P99 ITL (ms):                            125.45
==================================================

Итог: vLLM Средняя задержка = 5.32s, SGLang Средняя задержка = 25.15s. vLLM быстрее на 19.83 s (4.73x быстрее). TTFT = 68.48 ms против 6456.42 ms, снижение на 6387.94 ms (94.29x быстрее); TPOT = 2.35 ms против 89.37 ms, снижение на 87.02 ms (38.10x быстрее).

Quantization

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             42
Benchmark duration (s):                  130.13
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.68
Output token throughput (tok/s):         2194.80
Peak output token throughput (tok/s):    716367.44
Peak concurrent requests:                42.00
Total Token throughput (tok/s):          4862.60
----------------------Latency---------------------
Mean Latency(s):                          3.53
Median Latency(s):                        3.09
P95 Latency(s):                           8.98
P99 Latency(s):                           10.58
---------------Time to First Token----------------
Mean TTFT (ms):                          51.56
Median TTFT (ms):                        43.82
P95 TTFT (ms):                           91.52
P99 TTFT (ms):                           107.55
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.61
Median TPOT (ms):                        0.14
P95 TPOT (ms):                           12.48
P99 TPOT (ms):                           13.57
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.43
Median ITL (ms):                         0.00
P95 ITL (ms):                            12.41
P99 ITL (ms):                            13.52
==================================================

CUDA graph и Quantization

Размер 256

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--max-cudagraph-capture-size=256
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             43
Benchmark duration (s):                  130.15
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.68
Output token throughput (tok/s):         2179.71
Peak output token throughput (tok/s):    634473.24
Peak concurrent requests:                43.00
Total Token throughput (tok/s):          4829.15
----------------------Latency---------------------
Mean Latency(s):                          3.66
Median Latency(s):                        3.21
P95 Latency(s):                           9.22
P99 Latency(s):                           11.00
---------------Time to First Token----------------
Mean TTFT (ms):                          55.07
Median TTFT (ms):                        44.14
P95 TTFT (ms):                           94.53
P99 TTFT (ms):                           109.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.54
Median TPOT (ms):                        0.15
P95 TPOT (ms):                           12.82
P99 TPOT (ms):                           13.89
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.35
Median ITL (ms):                         0.00
P95 ITL (ms):                            12.71
P99 ITL (ms):                            13.81
==================================================

Размер 512

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--max-cudagraph-capture-size=512
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             42
Benchmark duration (s):                  130.15
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.68
Output token throughput (tok/s):         2180.27
Peak output token throughput (tok/s):    2532773.65
Peak concurrent requests:                42.00
Total Token throughput (tok/s):          4830.39
----------------------Latency---------------------
Mean Latency(s):                          3.54
Median Latency(s):                        3.11
P95 Latency(s):                           8.97
P99 Latency(s):                           10.69
---------------Time to First Token----------------
Mean TTFT (ms):                          51.99
Median TTFT (ms):                        44.02
P95 TTFT (ms):                           92.66
P99 TTFT (ms):                           110.20
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.77
Median TPOT (ms):                        0.15
P95 TPOT (ms):                           12.69
P99 TPOT (ms):                           13.91
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.59
Median ITL (ms):                         0.00
P95 ITL (ms):                            12.59
P99 ITL (ms):                            13.85
==================================================

Итог: Size 512 Средняя задержка = 3.54s, Size 256 Средняя задержка = 3.66s. Size 512 быстрее на 0.12 s (1.03x быстрее). TTFT = 51.99 ms против 55.07 ms, снижение на 3.07 ms (1.06x быстрее); TPOT = 1.77 ms против 1.54 ms, увеличение на 0.23 ms (1.15x медленнее).

Максимум токенов в батче и Quantization

16k

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-batched-tokens=16384

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             59
Benchmark duration (s):                  130.18
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.68
Output token throughput (tok/s):         2179.90
Peak output token throughput (tok/s):    13119091.07
Peak concurrent requests:                59.00
Total Token throughput (tok/s):          4829.57
----------------------Latency---------------------
Mean Latency(s):                          3.82
Median Latency(s):                        3.34
P95 Latency(s):                           9.63
P99 Latency(s):                           11.31
---------------Time to First Token----------------
Mean TTFT (ms):                          179.88
Median TTFT (ms):                        44.93
P95 TTFT (ms):                           262.90
P99 TTFT (ms):                           4297.33
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.28
Median TPOT (ms):                        0.15
P95 TPOT (ms):                           13.17
P99 TPOT (ms):                           19.61
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.65
Median ITL (ms):                         0.00
P95 ITL (ms):                            12.74
P99 ITL (ms):                            14.42
==================================================

32k

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-batched-tokens=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             57
Benchmark duration (s):                  130.14
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.68
Output token throughput (tok/s):         2216.28
Peak output token throughput (tok/s):    17112760.32
Peak concurrent requests:                57.00
Total Token throughput (tok/s):          4910.18
----------------------Latency---------------------
Mean Latency(s):                          3.82
Median Latency(s):                        3.36
P95 Latency(s):                           9.55
P99 Latency(s):                           11.33
---------------Time to First Token----------------
Mean TTFT (ms):                          198.01
Median TTFT (ms):                        44.82
P95 TTFT (ms):                           239.63
P99 TTFT (ms):                           4772.71
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.26
Median TPOT (ms):                        0.15
P95 TPOT (ms):                           13.35
P99 TPOT (ms):                           21.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.56
Median ITL (ms):                         0.00
P95 ITL (ms):                            13.02
P99 ITL (ms):                            14.55
==================================================

Итог: 32k Средняя задержка = 3.82s, 16k Средняя задержка = 3.82s. 32k быстрее на 0.00 s (1.00x быстрее). TTFT = 198.01 ms против 179.88 ms, увеличение на 18.12 ms (1.10x медленнее); TPOT = 2.26 ms против 2.28 ms, снижение на 0.02 ms (1.01x быстрее).

Режим производительности и Quantization

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--performance-mode=interactivity
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             55
Benchmark duration (s):                  129.88
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.70
Output token throughput (tok/s):         2187.29
Peak output token throughput (tok/s):    10792660.29
Peak concurrent requests:                55.00
Total Token throughput (tok/s):          4845.96
----------------------Latency---------------------
Mean Latency(s):                          3.67
Median Latency(s):                        3.25
P95 Latency(s):                           9.10
P99 Latency(s):                           11.00
---------------Time to First Token----------------
Mean TTFT (ms):                          184.63
Median TTFT (ms):                        43.80
P95 TTFT (ms):                           311.58
P99 TTFT (ms):                           4340.84
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.16
Median TPOT (ms):                        0.15
P95 TPOT (ms):                           13.05
P99 TPOT (ms):                           20.02
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.51
Median ITL (ms):                         0.00
P95 ITL (ms):                            12.37
P99 ITL (ms):                            15.66
==================================================

Кэш префиксов и Quantization

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768
--enable-prefix-caching

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             44
Benchmark duration (s):                  130.20
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.68
Output token throughput (tok/s):         2179.34
Peak output token throughput (tok/s):    1595392.00
Peak concurrent requests:                44.00
Total Token throughput (tok/s):          4828.33
----------------------Latency---------------------
Mean Latency(s):                          3.70
Median Latency(s):                        3.24
P95 Latency(s):                           9.43
P99 Latency(s):                           11.34
---------------Time to First Token----------------
Mean TTFT (ms):                          58.68
Median TTFT (ms):                        44.61
P95 TTFT (ms):                           125.39
P99 TTFT (ms):                           249.12
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.81
Median TPOT (ms):                        0.15
P95 TPOT (ms):                           12.83
P99 TPOT (ms):                           14.84
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.60
Median ITL (ms):                         0.00
P95 ITL (ms):                            12.75
P99 ITL (ms):                            14.80
==================================================

Спекулятивное декодирование и Quantization

Профиль: ShareGPT 8RPS

Параметры бэкенда:

--reasoning-parser=qwen3
--speculative-config={"method":"mtp","num_speculative_tokens":1}
--max-model-len=32768

Результат бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             57
Benchmark duration (s):                  129.19
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.74
Output token throughput (tok/s):         2203.31
Peak output token throughput (tok/s):    8794775.78
Peak concurrent requests:                57.00
Total Token throughput (tok/s):          4881.45
----------------------Latency---------------------
Mean Latency(s):                          3.28
Median Latency(s):                        2.85
P95 Latency(s):                           8.22
P99 Latency(s):                           10.49
---------------Time to First Token----------------
Mean TTFT (ms):                          263.11
Median TTFT (ms):                        60.45
P95 TTFT (ms):                           891.65
P99 TTFT (ms):                           5402.99
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.34
Median TPOT (ms):                        0.21
P95 TPOT (ms):                           11.39
P99 TPOT (ms):                           22.27
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.41
Median ITL (ms):                         0.00
P95 ITL (ms):                            10.58
P99 ITL (ms):                            14.34
==================================================

Сводка методов оптимизации

Сценарий бенчмарка	Группа	После оптимизации	Базовая конфигурация	Сравнение
ShareGPT 8RPS (r=8)	Выбор движка инференса	5.32s	5.32s	(1.00x faster)
ShareGPT 8RPS (r=8)	Quantization	3.53s	5.32s	(1.51x faster)
ShareGPT 8RPS (r=8)	CUDA graph && Quantization	3.54s	5.32s	(1.50x faster)
ShareGPT 8RPS (r=8)	Максимум токенов в батче && Quantization	3.82s	5.32s	(1.39x faster)
ShareGPT 8RPS (r=8)	Режим производительности && Quantization	3.67s	5.32s	(1.45x faster)
ShareGPT 8RPS (r=8)	Кэш префиксов && Quantization	3.70s	5.32s	(1.44x faster)
ShareGPT 8RPS (r=8)	Спекулятивное декодирование && Quantization	3.28s	5.32s	(1.63x faster)

Другие сценарии бенчмарка

Частота запросов 1

Базовые параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат базового бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             6
Benchmark duration (s):                  1000.76
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              1.00
Output token throughput (tok/s):         281.70
Peak output token throughput (tok/s):    205535.27
Peak concurrent requests:                6.00
Total Token throughput (tok/s):          624.11
----------------------Latency---------------------
Mean Latency(s):                          1.85
Median Latency(s):                        1.57
P95 Latency(s):                           4.80
P99 Latency(s):                           5.90
---------------Time to First Token----------------
Mean TTFT (ms):                          46.01
Median TTFT (ms):                        35.63
P95 TTFT (ms):                           90.77
P99 TTFT (ms):                           98.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.06
Median TPOT (ms):                        0.12
P95 TPOT (ms):                           6.90
P99 TPOT (ms):                           7.56
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.90
Median ITL (ms):                         0.00
P95 ITL (ms):                            6.81
P99 ITL (ms):                            7.50
==================================================

Параметры бэкенда после оптимизации:

--reasoning-parser=qwen3
--speculative-config={"method":"mtp","num_speculative_tokens":1}
--max-model-len=32768

Результат бенчмарка после оптимизации

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             5
Benchmark duration (s):                  1000.01
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              1.00
Output token throughput (tok/s):         281.71
Peak output token throughput (tok/s):    160685.52
Peak concurrent requests:                5.00
Total Token throughput (tok/s):          624.13
----------------------Latency---------------------
Mean Latency(s):                          1.50
Median Latency(s):                        1.31
P95 Latency(s):                           3.88
P99 Latency(s):                           4.65
---------------Time to First Token----------------
Mean TTFT (ms):                          66.56
Median TTFT (ms):                        55.47
P95 TTFT (ms):                           128.15
P99 TTFT (ms):                           145.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.89
Median TPOT (ms):                        0.17
P95 TPOT (ms):                           5.38
P99 TPOT (ms):                           5.89
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.66
Median ITL (ms):                         0.00
P95 ITL (ms):                            5.25
P99 ITL (ms):                            5.75
==================================================

Частота запросов 4

Базовые параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат базового бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             34
Benchmark duration (s):                  254.48
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              3.93
Output token throughput (tok/s):         1119.46
Peak output token throughput (tok/s):    12576036.09
Peak concurrent requests:                34.00
Total Token throughput (tok/s):          2480.18
----------------------Latency---------------------
Mean Latency(s):                          3.30
Median Latency(s):                        2.84
P95 Latency(s):                           8.64
P99 Latency(s):                           10.14
---------------Time to First Token----------------
Mean TTFT (ms):                          126.16
Median TTFT (ms):                        42.08
P95 TTFT (ms):                           96.81
P99 TTFT (ms):                           3590.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.02
Median TPOT (ms):                        0.14
P95 TPOT (ms):                           11.71
P99 TPOT (ms):                           15.87
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.57
Median ITL (ms):                         0.00
P95 ITL (ms):                            11.49
P99 ITL (ms):                            13.24
==================================================

Параметры бэкенда после оптимизации:

--reasoning-parser=qwen3
--speculative-config={"method":"mtp","num_speculative_tokens":1}
--max-model-len=32768

Результат бенчмарка после оптимизации

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             35
Benchmark duration (s):                  253.07
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              3.95
Output token throughput (tok/s):         1126.78
Peak output token throughput (tok/s):    14184016.34
Peak concurrent requests:                35.00
Total Token throughput (tok/s):          2496.38
----------------------Latency---------------------
Mean Latency(s):                          2.52
Median Latency(s):                        2.13
P95 Latency(s):                           6.48
P99 Latency(s):                           10.07
---------------Time to First Token----------------
Mean TTFT (ms):                          221.71
Median TTFT (ms):                        62.58
P95 TTFT (ms):                           143.76
P99 TTFT (ms):                           5977.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.78
Median TPOT (ms):                        0.21
P95 TPOT (ms):                           8.39
P99 TPOT (ms):                           20.07
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.99
Median ITL (ms):                         0.00
P95 ITL (ms):                            7.98
P99 ITL (ms):                            9.26
==================================================

Частота запросов 8

Базовые параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат базового бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             59
Benchmark duration (s):                  132.25
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.56
Output token throughput (tok/s):         2137.15
Peak output token throughput (tok/s):    1410355.61
Peak concurrent requests:                59.00
Total Token throughput (tok/s):          4734.87
----------------------Latency---------------------
Mean Latency(s):                          5.32
Median Latency(s):                        4.72
P95 Latency(s):                           13.39
P99 Latency(s):                           16.27
---------------Time to First Token----------------
Mean TTFT (ms):                          68.48
Median TTFT (ms):                        62.59
P95 TTFT (ms):                           102.52
P99 TTFT (ms):                           131.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.35
Median TPOT (ms):                        0.20
P95 TPOT (ms):                           18.88
P99 TPOT (ms):                           20.04
---------------Inter-token Latency----------------
Mean ITL (ms):                           2.11
Median ITL (ms):                         0.00
P95 ITL (ms):                            18.78
P99 ITL (ms):                            19.94
==================================================

Параметры бэкенда после оптимизации:

--reasoning-parser=qwen3
--speculative-config={"method":"mtp","num_speculative_tokens":1}
--max-model-len=32768

Результат бенчмарка после оптимизации

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             57
Benchmark duration (s):                  129.19
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              7.74
Output token throughput (tok/s):         2203.31
Peak output token throughput (tok/s):    8794775.78
Peak concurrent requests:                57.00
Total Token throughput (tok/s):          4881.45
----------------------Latency---------------------
Mean Latency(s):                          3.28
Median Latency(s):                        2.85
P95 Latency(s):                           8.22
P99 Latency(s):                           10.49
---------------Time to First Token----------------
Mean TTFT (ms):                          263.11
Median TTFT (ms):                        60.45
P95 TTFT (ms):                           891.65
P99 TTFT (ms):                           5402.99
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.34
Median TPOT (ms):                        0.21
P95 TPOT (ms):                           11.39
P99 TPOT (ms):                           22.27
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.41
Median ITL (ms):                         0.00
P95 ITL (ms):                            10.58
P99 ITL (ms):                            14.34
==================================================

Частота запросов 16

Базовые параметры бэкенда:

--reasoning-parser=qwen3
--max-model-len=32768

Результат базового бенчмарка

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             447
Benchmark duration (s):                  80.77
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              12.38
Output token throughput (tok/s):         3499.91
Peak output token throughput (tok/s):    4256990.61
Peak concurrent requests:                447.00
Total Token throughput (tok/s):          7754.06
----------------------Latency---------------------
Mean Latency(s):                          18.59
Median Latency(s):                        16.18
P95 Latency(s):                           43.61
P99 Latency(s):                           53.53
---------------Time to First Token----------------
Mean TTFT (ms):                          232.24
Median TTFT (ms):                        229.86
P95 TTFT (ms):                           402.69
P99 TTFT (ms):                           465.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.90
Median TPOT (ms):                        0.67
P95 TPOT (ms):                           68.77
P99 TPOT (ms):                           94.53
---------------Inter-token Latency----------------
Mean ITL (ms):                           8.11
Median ITL (ms):                         0.00
P95 ITL (ms):                            68.39
P99 ITL (ms):                            93.98
==================================================

Параметры бэкенда после оптимизации:

--reasoning-parser=qwen3
--speculative-config={"method":"mtp","num_speculative_tokens":1}
--max-model-len=32768

Результат бенчмарка после оптимизации

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             145
Benchmark duration (s):                  69.63
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              14.36
Output token throughput (tok/s):         4070.93
Peak output token throughput (tok/s):    3898930.48
Peak concurrent requests:                145.00
Total Token throughput (tok/s):          9019.16
----------------------Latency---------------------
Mean Latency(s):                          6.79
Median Latency(s):                        5.91
P95 Latency(s):                           16.32
P99 Latency(s):                           21.54
---------------Time to First Token----------------
Mean TTFT (ms):                          150.30
Median TTFT (ms):                        133.22
P95 TTFT (ms):                           293.44
P99 TTFT (ms):                           345.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.23
Median TPOT (ms):                        0.42
P95 TPOT (ms):                           25.40
P99 TPOT (ms):                           28.30
---------------Inter-token Latency----------------
Mean ITL (ms):                           2.70
Median ITL (ms):                         0.00
P95 ITL (ms):                            25.13
P99 ITL (ms):                            28.02
==================================================