Qwen3.5-9B 吞吐量优化

结论

Qwen/Qwen3.5-9B 在 NVIDIA H100 80GB HBM3 上的吞吐量优化推荐配置：

推理服务命令

vllm serve Qwen/Qwen3.5-9B \
    --reasoning-parser=qwen3 \
    --performance-mode=throughput \
    --max-model-len=32768 \
    --max-num-seqs=512

优化前后的基准测试结果对比：

基准测试用例	基准配置（未做任何优化的 vLLM）	优化后
ShareGPT 测试配置	Total TPS: 11527.14 Mean TPOT(ms): 15.12	Total TPS: 11768.22 (+2.09%) Mean TPOT(ms): 15.85
吞吐量测试配置	Total TPS: 29822.16 Mean TPOT(ms): 34.63	Total TPS: 34470.14 (+15.59%) Mean TPOT(ms): 36.97
长上下文测试配置	Total TPS: 32416.84 Mean TPOT(ms): 10.21	Total TPS: 32658.73 (+0.75%) Mean TPOT(ms): 9.97
生成密集型测试配置	Total TPS: 2973.42 Mean TPOT(ms): 0.33	Total TPS: 3025.11 (+1.74%) Mean TPOT(ms): 0.12

说明

基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph 等。本文结论可作为进一步针对性优化的起点。
测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。
量化可能影响准确率，但对于大多数模型，FP8 量化的准确率下降可控制在 1% 以内。详情参见评估结果。因此，强烈建议在低延迟推理服务场景中使用 FP8 量化。
推测解码能够显著降低低并发请求的延迟，但加速效果取决于基准测试数据集的数据分布和草稿模型的选择。例如，本测试所选的草稿模型使用英文数据训练，在其他语言上的表现可能不够理想。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

实验环境

模型

Qwen/Qwen3.5-9B

硬件

NVIDIA H100 80GB HBM3

引擎版本

vLLM v0.17.1
SGLang v0.5.9

基准测试方法

本项目使用 GPUStack 的一键基准测试功能评估推理服务工作负载。本文中的基准测试均通过该流程执行。

GPUStack 的基准测试通过封装项目 benchmark-runner，基于 guidellm 实现。

对于以下基准测试配置，GPUStack 负责模型部署、基准测试任务提交和结果收集。

基准测试配置

ShareGPT

dataset_name: ShareGPT
request_rate: 1000
total_requests: 1000

吞吐量

dataset_name: Random
dataset_input_tokens: 1024
dataset_output_tokens: 128
dataset_seed: 42
request_rate: 1000
total_requests: 1000

长上下文

dataset_name: Random
dataset_input_tokens: 32000
dataset_output_tokens: 100
dataset_seed: 42
request_rate: 1
total_requests: 100

生成密集型

dataset_name: Random
dataset_input_tokens: 1000
dataset_output_tokens: 2000
dataset_seed: 42
request_rate: 1
total_requests: 200

实验结果

选择推理引擎

vLLM

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--max-model-len=32768

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.74
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.27
Output token throughput (tok/s):         5202.94
Peak output token throughput (tok/s):    20132659.20
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11527.14
----------------------Latency---------------------
Mean Latency(s):                          24.11
Median Latency(s):                        21.90
P99 Latency(s):                           51.16
---------------Time to First Token----------------
Mean TTFT (ms):                          1604.23
Median TTFT (ms):                        646.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5737.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.12
Median TPOT (ms):                        3.42
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.68
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.45
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            118.73
==================================================

SGLang

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--context-length=32768

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  96.01
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              10.42
Output token throughput (tok/s):         2933.60
Peak output token throughput (tok/s):    92831.68
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          6499.41
----------------------Latency---------------------
Mean Latency(s):                          39.08
Median Latency(s):                        41.18
P99 Latency(s):                           74.40
---------------Time to First Token----------------
Mean TTFT (ms):                          26217.15
Median TTFT (ms):                        34401.34
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           41101.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          138.86
Median TPOT (ms):                        112.19
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           591.70
---------------Inter-token Latency----------------
Mean ITL (ms):                           45.86
Median ITL (ms):                         49.31
P95 ITL (ms):                            N/A
P99 ITL (ms):                            59.71
==================================================

汇总：vLLM Total TPS = 11527.14, SGLang Total TPS = 6499.41. vLLM 性能更优，差值为 5027.72 tok/s (77.36%); Mean TPOT = 15.12 ms 与 138.86 ms, 减少 123.74 ms (89.11%).

前缀缓存

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--enable-prefix-caching
--max-model-len=32768

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  56.47
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              17.71
Output token throughput (tok/s):         5066.42
Peak output token throughput (tok/s):    31242972.50
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11224.70
----------------------Latency---------------------
Mean Latency(s):                          24.90
Median Latency(s):                        22.40
P99 Latency(s):                           52.82
---------------Time to First Token----------------
Mean TTFT (ms):                          1995.45
Median TTFT (ms):                        1341.75
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           6549.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.66
Median TPOT (ms):                        5.21
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           130.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.60
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            120.90
==================================================

推测解码

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--speculative-config={"method":"mtp","num_speculative_tokens":4}
--max-model-len=32768

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     514
Maximum request concurrency:             512
Benchmark duration (s):                  12.27
Total input tokens:                      23313
Total generated tokens:                  131530
Request throughput (req/s):              41.90
Output token throughput (tok/s):         12792.49
Peak output token throughput (tok/s):    13918875.50
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          15059.89
----------------------Latency---------------------
Mean Latency(s):                          11.77
Median Latency(s):                        11.81
P99 Latency(s):                           12.07
---------------Time to First Token----------------
Mean TTFT (ms):                          268.95
Median TTFT (ms):                        0.00
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           3010.96
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          42.50
Median TPOT (ms):                        31.35
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           214.99
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

性能模式

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.66
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.29
Output token throughput (tok/s):         5189.06
Peak output token throughput (tok/s):    30482939.47
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11496.39
----------------------Latency---------------------
Mean Latency(s):                          24.00
Median Latency(s):                        21.51
P95 Latency(s):                           N/A
P99 Latency(s):                           51.01
---------------Time to First Token----------------
Mean TTFT (ms):                          1824.24
Median TTFT (ms):                        982.10
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5550.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.41
Median TPOT (ms):                        4.53
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.10
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.96
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            115.72
==================================================

最大序列数

序列数 512

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-seqs=512

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.89
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.22
Output token throughput (tok/s):         5174.68
Peak output token throughput (tok/s):    21197768.76
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11464.53
----------------------Latency---------------------
Mean Latency(s):                          24.11
Median Latency(s):                        21.96
P95 Latency(s):                           48.41
P99 Latency(s):                           51.17
---------------Time to First Token----------------
Mean TTFT (ms):                          1893.47
Median TTFT (ms):                        1233.85
P95 TTFT (ms):                           5659.91
P99 TTFT (ms):                           6222.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.18
Median TPOT (ms):                        4.48
P95 TPOT (ms):                           91.71
P99 TPOT (ms):                           127.60
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.48
Median ITL (ms):                         0.00
P95 ITL (ms):                            83.92
P99 ITL (ms):                            116.65
==================================================

序列数 1024

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-seqs=1024

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.87
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.23
Output token throughput (tok/s):         5205.42
Peak output token throughput (tok/s):    22885338.88
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11532.64
----------------------Latency---------------------
Mean Latency(s):                          24.15
Median Latency(s):                        21.71
P95 Latency(s):                           48.49
P99 Latency(s):                           51.24
---------------Time to First Token----------------
Mean TTFT (ms):                          1864.39
Median TTFT (ms):                        1137.51
P95 TTFT (ms):                           5803.37
P99 TTFT (ms):                           6125.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.87
Median TPOT (ms):                        4.70
P95 TPOT (ms):                           93.65
P99 TPOT (ms):                           127.45
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.28
Median ITL (ms):                         0.00
P95 ITL (ms):                            87.39
P99 ITL (ms):                            116.13
==================================================

序列数 2048

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-seqs=2048

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  55.28
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.09
Output token throughput (tok/s):         5140.19
Peak output token throughput (tok/s):    23553347.13
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11388.13
----------------------Latency---------------------
Mean Latency(s):                          24.34
Median Latency(s):                        22.16
P95 Latency(s):                           48.86
P99 Latency(s):                           51.63
---------------Time to First Token----------------
Mean TTFT (ms):                          1853.85
Median TTFT (ms):                        1166.25
P95 TTFT (ms):                           5543.84
P99 TTFT (ms):                           6126.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.91
Median TPOT (ms):                        4.35
P95 TPOT (ms):                           100.86
P99 TPOT (ms):                           129.26
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.36
Median ITL (ms):                         0.00
P95 ITL (ms):                            94.30
P99 ITL (ms):                            118.29
==================================================

汇总：Seqs 1024 Total TPS = 11532.64, Seqs 2048 Total TPS = 11388.13. Seqs 1024 性能更优，差值为 144.51 tok/s (1.27%); Mean TPOT = 16.87 ms 与 16.91 ms, 减少 0.04 ms (0.25%).

最大批处理 Token 数 && 性能模式

24k

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=24576

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.56
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.33
Output token throughput (tok/s):         5196.51
Peak output token throughput (tok/s):    32356059.43
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11512.90
----------------------Latency---------------------
Mean Latency(s):                          23.98
Median Latency(s):                        21.78
P99 Latency(s):                           50.95
---------------Time to First Token----------------
Mean TTFT (ms):                          1870.77
Median TTFT (ms):                        1771.31
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5297.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.83
Median TPOT (ms):                        4.86
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           125.06
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.22
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            113.92
==================================================

32k

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=32768

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.20
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.45
Output token throughput (tok/s):         5268.65
Peak output token throughput (tok/s):    29985425.61
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11672.72
----------------------Latency---------------------
Mean Latency(s):                          23.72
Median Latency(s):                        21.21
P99 Latency(s):                           50.44
---------------Time to First Token----------------
Mean TTFT (ms):                          1970.35
Median TTFT (ms):                        1344.71
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5206.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.53
Median TPOT (ms):                        5.04
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           125.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.56
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            111.59
==================================================

汇总：32k Total TPS = 11672.72, 24k Total TPS = 11512.90. 32k 性能更优，差值为 159.82 tok/s (1.39%); Mean TPOT = 16.53 ms 与 15.83 ms, 增加 0.70 ms (4.43%，耗时增加).

最大序列数 && 性能模式

最大序列数 256

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=256

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  57.32
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              17.45
Output token throughput (tok/s):         4973.66
Peak output token throughput (tok/s):    30062067.31
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11019.18
----------------------Latency---------------------
Mean Latency(s):                          23.57
Median Latency(s):                        22.86
P99 Latency(s):                           49.24
---------------Time to First Token----------------
Mean TTFT (ms):                          11061.67
Median TTFT (ms):                        14983.14
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           18807.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          44.58
Median TPOT (ms):                        31.66
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           229.02
---------------Inter-token Latency----------------
Mean ITL (ms):                           5.29
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            54.12
==================================================

最大序列数 512

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  53.72
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.62
Output token throughput (tok/s):         5311.75
Peak output token throughput (tok/s):    23159852.52
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11768.22
----------------------Latency---------------------
Mean Latency(s):                          23.50
Median Latency(s):                        21.13
P99 Latency(s):                           50.04
---------------Time to First Token----------------
Mean TTFT (ms):                          1826.53
Median TTFT (ms):                        1004.67
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5582.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.85
Median TPOT (ms):                        4.49
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           123.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.39
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            112.55
==================================================

汇总：Max Num Seqs 512 Total TPS = 11768.22, Max Num Seqs 256 Total TPS = 11019.18. Max Num Seqs 512 性能更优，差值为 749.05 tok/s (6.80%); Mean TPOT = 15.85 ms 与 44.58 ms, 减少 28.73 ms (64.45%).

最大序列数 && 最大批处理 Token 数

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=32768
--max-num-seqs=512

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  55.33
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.07
Output token throughput (tok/s):         5115.71
Peak output token throughput (tok/s):    34751436.57
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11333.89
----------------------Latency---------------------
Mean Latency(s):                          24.34
Median Latency(s):                        22.08
P99 Latency(s):                           51.69
---------------Time to First Token----------------
Mean TTFT (ms):                          1831.81
Median TTFT (ms):                        1345.90
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5161.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.39
Median TPOT (ms):                        4.48
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.14
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.92
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            116.48
==================================================

最大序列数 && 最大批处理 Token 数 && 性能模式

批处理 Token 数 32k，序列数 512

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=32768
--max-num-seqs=512

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  55.33
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.07
Output token throughput (tok/s):         5115.71
Peak output token throughput (tok/s):    34751436.57
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11333.89
----------------------Latency---------------------
Mean Latency(s):                          24.34
Median Latency(s):                        22.08
P95 Latency(s):                           N/A
P99 Latency(s):                           51.69
---------------Time to First Token----------------
Mean TTFT (ms):                          1831.81
Median TTFT (ms):                        1345.90
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5161.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.39
Median TPOT (ms):                        4.48
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.14
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.92
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            116.48
==================================================

批处理 Token 数 48k，序列数 512

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=49152
--max-num-seqs=512

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.48
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.36
Output token throughput (tok/s):         5223.82
Peak output token throughput (tok/s):    33459752.45
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11573.41
----------------------Latency---------------------
Mean Latency(s):                          23.96
Median Latency(s):                        21.63
P95 Latency(s):                           N/A
P99 Latency(s):                           50.92
---------------Time to First Token----------------
Mean TTFT (ms):                          2022.91
Median TTFT (ms):                        2480.42
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5146.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.20
Median TPOT (ms):                        5.63
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           124.01
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.04
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            111.00
==================================================

批处理 Token 数 48k，序列数 768

测试配置：ShareGPT

后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=49152
--max-num-seqs=768

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.07
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.49
Output token throughput (tok/s):         5248.67
Peak output token throughput (tok/s):    36318641.50
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11628.47
----------------------Latency---------------------
Mean Latency(s):                          23.72
Median Latency(s):                        21.75
P95 Latency(s):                           N/A
P99 Latency(s):                           50.48
---------------Time to First Token----------------
Mean TTFT (ms):                          1817.87
Median TTFT (ms):                        1821.59
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5041.22
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.65
Median TPOT (ms):                        4.57
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           123.59
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.23
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            114.09
==================================================

汇总：Batch Token 48k and Seqs 768 Total TPS = 11628.47, Batch Token 32k and Seqs 512 Total TPS = 11333.89. Batch Token 48k and Seqs 768 性能更优，差值为 294.57 tok/s (2.60%); Mean TPOT = 15.65 ms 与 16.39 ms, 减少 0.74 ms (4.51%).

优化选项汇总

基准测试用例	优化后	基准配置
选择推理引擎	Total TPS: 11527.14 (+0.00%) Mean TPOT(ms): 15.12	Total TPS: 11527.14 Mean TPOT(ms): 15.12
前缀缓存	Total TPS: 11224.70 (-2.62%) Mean TPOT(ms): 16.66	Total TPS: 11527.14 Mean TPOT(ms): 15.12
推测解码	Total TPS: 15059.89 (成功率：51.5%, 不采用此优化) Mean TPOT(ms): 42.50	Total TPS: 11527.14 Mean TPOT(ms): 15.12
性能模式	Total TPS: 11496.39 (-0.27%) Mean TPOT(ms): 16.41	Total TPS: 11527.14 Mean TPOT(ms): 15.12
最大序列数	Total TPS: 11532.64 (+0.05%) Mean TPOT(ms): 16.87	Total TPS: 11527.14 Mean TPOT(ms): 15.12
最大批处理 Token 数 && 性能模式	Total TPS: 11672.72 (+1.26%) Mean TPOT(ms): 16.53	Total TPS: 11527.14 Mean TPOT(ms): 15.12
最大序列数 && 性能模式	Total TPS: 11768.22 (+2.09%) Mean TPOT(ms): 15.85	Total TPS: 11527.14 Mean TPOT(ms): 15.12
最大序列数 && 最大批处理 Token 数	Total TPS: 11333.89 (-1.68%) Mean TPOT(ms): 16.39	Total TPS: 11527.14 Mean TPOT(ms): 15.12
最大序列数 && 最大批处理 Token 数 && 性能模式	Total TPS: 11628.47 (+0.88%) Mean TPOT(ms): 15.65	Total TPS: 11527.14 Mean TPOT(ms): 15.12

其他基准测试用例

ShareGPT 测试配置

基准配置的后端参数：

--reasoning-parser=qwen3
--max-model-len=32768

基准配置的测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.74
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.27
Output token throughput (tok/s):         5202.94
Peak output token throughput (tok/s):    20132659.20
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11527.14
----------------------Latency---------------------
Mean Latency(s):                          24.11
Median Latency(s):                        21.90
P99 Latency(s):                           51.16
---------------Time to First Token----------------
Mean TTFT (ms):                          1604.23
Median TTFT (ms):                        646.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5737.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.12
Median TPOT (ms):                        3.42
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.68
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.45
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            118.73
==================================================

优化后的后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

优化后的测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  53.72
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.62
Output token throughput (tok/s):         5311.75
Peak output token throughput (tok/s):    23159852.52
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11768.22
----------------------Latency---------------------
Mean Latency(s):                          23.50
Median Latency(s):                        21.13
P99 Latency(s):                           50.04
---------------Time to First Token----------------
Mean TTFT (ms):                          1826.53
Median TTFT (ms):                        1004.67
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5582.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.85
Median TPOT (ms):                        4.49
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           123.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.39
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            112.55
==================================================

吞吐量测试配置

基准配置的后端参数：

--reasoning-parser=qwen3
--max-model-len=32768

基准配置的测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  43.00
Total input tokens:                      1033932
Total generated tokens:                  128000
Request throughput (req/s):              23.26
Output token throughput (tok/s):         3285.25
Peak output token throughput (tok/s):    4418690.63
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          29822.16
----------------------Latency---------------------
Mean Latency(s):                          21.27
Median Latency(s):                        21.11
P99 Latency(s):                           36.40
---------------Time to First Token----------------
Mean TTFT (ms):                          4433.02
Median TTFT (ms):                        1510.44
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           14914.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          34.63
Median TPOT (ms):                        11.80
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           116.52
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

优化后的后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

优化后的测试结果

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  39.10
Total input tokens:                      1033932
Total generated tokens:                  128000
Request throughput (req/s):              25.55
Output token throughput (tok/s):         3797.28
Peak output token throughput (tok/s):    7724761.32
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          34470.14
----------------------Latency---------------------
Mean Latency(s):                          19.38
Median Latency(s):                        19.24
P99 Latency(s):                           31.36
---------------Time to First Token----------------
Mean TTFT (ms):                          4731.56
Median TTFT (ms):                        2442.61
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           13309.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          36.97
Median TPOT (ms):                        19.08
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           103.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

长上下文测试配置

基准配置的后端参数：

--reasoning-parser=qwen3
--max-model-len=32768

基准配置的测试结果

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             9
Benchmark duration (s):                  100.89
Total input tokens:                      3200993
Total generated tokens:                  10000
Request throughput (req/s):              0.99
Output token throughput (tok/s):         100.96
Peak output token throughput (tok/s):    198.08
Peak concurrent requests:                9.00
Total Token throughput (tok/s):          32416.84
----------------------Latency---------------------
Mean Latency(s):                          7.88
Median Latency(s):                        8.10
P95 Latency(s):                           N/A
P99 Latency(s):                           8.54
---------------Time to First Token----------------
Mean TTFT (ms):                          1021.12
Median TTFT (ms):                        1020.12
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           1041.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.21
Median TPOT (ms):                        10.20
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           10.42
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

优化后的后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

优化后的测试结果

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             14
Benchmark duration (s):                  100.13
Total input tokens:                      3200993
Total generated tokens:                  10000
Request throughput (req/s):              1.00
Output token throughput (tok/s):         101.71
Peak output token throughput (tok/s):    227.24
Peak concurrent requests:                14.00
Total Token throughput (tok/s):          32658.73
----------------------Latency---------------------
Mean Latency(s):                          8.00
Median Latency(s):                        8.00
P99 Latency(s):                           12.36
---------------Time to First Token----------------
Mean TTFT (ms):                          997.50
Median TTFT (ms):                        981.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           1421.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.97
Median TPOT (ms):                        9.81
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           14.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

生成密集型测试配置

基准配置的后端参数：

--reasoning-parser=qwen3
--max-model-len=32768

基准配置的测试结果

============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             18
Benchmark duration (s):                  214.96
Total input tokens:                      201982
Total generated tokens:                  400000
Request throughput (req/s):              0.93
Output token throughput (tok/s):         1975.75
Peak output token throughput (tok/s):    123795.17
Peak concurrent requests:                18.00
Total Token throughput (tok/s):          2973.42
----------------------Latency---------------------
Mean Latency(s):                          17.48
Median Latency(s):                        17.59
P99 Latency(s):                           17.60
---------------Time to First Token----------------
Mean TTFT (ms):                          54.95
Median TTFT (ms):                        55.17
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           59.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.33
Median TPOT (ms):                        0.03
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           8.79
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.31
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            8.77
==================================================

优化后的后端参数：

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

优化后的测试结果

============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             18
Benchmark duration (s):                  214.98
Total input tokens:                      201982
Total generated tokens:                  400000
Request throughput (req/s):              0.93
Output token throughput (tok/s):         2010.10
Peak output token throughput (tok/s):    59750.47
Peak concurrent requests:                18.00
Total Token throughput (tok/s):          3025.11
----------------------Latency---------------------
Mean Latency(s):                          17.55
Median Latency(s):                        17.66
P99 Latency(s):                           17.67
---------------Time to First Token----------------
Mean TTFT (ms):                          55.85
Median TTFT (ms):                        56.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           60.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.12
Median TPOT (ms):                        0.03
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           0.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.09
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================