跳转至

NVIDIA H100 GPUs 上的 Qwen3-32B 吞吐量优化

结论

qwen3-32b-h100

Qwen3-32B 在 H100 GPUs 上的吞吐量优化推荐配置：

推理服务命令 
1
trtllm-serve Qwen/Qwen3-32B-FP8 --max_seq_len=40960 --enable_chunked_prefill

优化前后的基准测试结果对比：

基准测试用例 基准配置（未做任何优化的 vLLM） 优化后
ShareGPT Total TPS: 2352.82
Mean TPOT(ms): 88.09		 Total TPS: 4284.68 (+82.1%)
Mean TPOT(ms): 158.64
短提示词 Total TPS: 4005.62
Mean TPOT(ms): 1169.72		 Total TPS: 6826.33 (+70.4%)
Mean TPOT(ms): 890.11
中等长度提示词 Total TPS: 3195.30
Mean TPOT(ms): 86.62		 Total TPS: 5901.95 (+84.7%)
Mean TPOT(ms): 223.40
长提示词 Total TPS: 2148.69
Mean TPOT(ms): 66.61		 Total TPS: 4929.16 (+129.4%)
Mean TPOT(ms): 138.75
超长提示词 Total TPS: 2771.81
Mean TPOT(ms): 43.42		 Total TPS: 4068.16 (+10.7%)
Mean TPOT(ms): 286.97

说明

  1. 基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
  2. 其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph、Torch Compile 等。本文结论可作为进一步针对性优化的起点。
  3. 测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。
  4. 量化可能影响准确率，但对于大多数模型，FP8 量化的准确率下降可控制在 1% 以内。详情参见评估结果。因此，强烈建议在高吞吐量推理服务场景中使用 FP8 量化。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

优化目标

在高并发请求场景下实现高吞吐量。

实验环境

模型

Qwen3-32B

硬件

单个 NVIDIA H100 SXM GPU。

引擎版本

  • vLLM: v0.11.0
  • SGLang: v0.5.5.post1
  • TensorRT-LLM: v1.2.0rc1

基准测试数据集

  1. ShareGPT
  2. 不同序列长度的随机数据集：
    • 超长提示词：32000 输入 tokens，100 输出 tokens
    • 长提示词：4000 输入 tokens，200 输出 tokens
    • 中等长度提示词：2000 输入 tokens，100 输出 tokens
    • 短提示词：128 输入 tokens，4 输出 tokens

基准测试脚本

使用 vLLM bench CLI 工具对模型性能进行基准测试。运行基准测试的命令如下：

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-32B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-32B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

实验结果

1. 选择推理引擎

vLLM

推理服务脚本 
1
vllm serve Qwen/Qwen3-32B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  178.01
Total input tokens:                      217393
Total generated tokens:                  201422
Request throughput (req/s):              5.62
Output token throughput (tok/s):         1131.55
Peak output token throughput (tok/s):    2360.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2352.82
---------------Time to First Token----------------
Mean TTFT (ms):                          78118.74
Median TTFT (ms):                        77291.46
P99 TTFT (ms):                           153161.51
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          88.09
Median TPOT (ms):                        70.72
P99 TPOT (ms):                           378.18
---------------Inter-token Latency----------------
Mean ITL (ms):                           69.76
Median ITL (ms):                         50.20
P99 ITL (ms):                            277.93
==================================================

SGLang

推理服务脚本 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-32B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  247.94
Total input tokens:                      217393
Total generated tokens:                  201826
Request throughput (req/s):              4.03
Output token throughput (tok/s):         814.03
Peak output token throughput (tok/s):    1329.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          1690.84
---------------Time to First Token----------------
Mean TTFT (ms):                          115690.30
Median TTFT (ms):                        117738.49
P99 TTFT (ms):                           230503.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          57.86
Median TPOT (ms):                        54.19
P99 TPOT (ms):                           95.84
---------------Inter-token Latency----------------
Mean ITL (ms):                           56.25
Median ITL (ms):                         43.94
P99 ITL (ms):                            207.79
==================================================

TensorRT-LLM

推理服务脚本 
1
trtllm-serve Qwen/Qwen3-32B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  157.59
Total input tokens:                      217393
Total generated tokens:                  201723
Request throughput (req/s):              6.35
Output token throughput (tok/s):         1280.04
Peak output token throughput (tok/s):    1958.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2659.50
---------------Time to First Token----------------
Mean TTFT (ms):                          65281.52
Median TTFT (ms):                        65325.76
P99 TTFT (ms):                           133106.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          61.90
Median TPOT (ms):                        59.59
P99 TPOT (ms):                           121.97
---------------Inter-token Latency----------------
Mean ITL (ms):                           59.22
Median ITL (ms):                         47.89
P99 ITL (ms):                            214.01
==================================================

结果：TensorRT-LLM (2659.50 tok/s) > vLLM (2352.82 tok/s) > SGLang (1690.84 tok/s)

2. TensorRT-LLM 中的 量化

FP8

推理服务脚本 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-32B-FP8
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  104.55
Total input tokens:                      217393
Total generated tokens:                  201376
Request throughput (req/s):              9.56
Output token throughput (tok/s):         1926.06
Peak output token throughput (tok/s):    4391.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4005.32
---------------Time to First Token----------------
Mean TTFT (ms):                          35510.55
Median TTFT (ms):                        31577.26
P99 TTFT (ms):                           79653.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          205.13
Median TPOT (ms):                        119.96
P99 TPOT (ms):                           1141.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           118.32
Median ITL (ms):                         86.53
P99 ITL (ms):                            1195.28
==================================================

3. 在 TensorRT-LLM 中启用分块预填充

推理服务脚本 
1
trtllm-serve Qwen/Qwen3-32B-FP8 --max_seq_len=40960 --enable_chunked_prefill
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  97.74
Total input tokens:                      217393
Total generated tokens:                  201376
Request throughput (req/s):              10.23
Output token throughput (tok/s):         2060.40
Peak output token throughput (tok/s):    4371.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4284.68
---------------Time to First Token----------------
Mean TTFT (ms):                          31208.75
Median TTFT (ms):                        25003.73
P99 TTFT (ms):                           72827.99
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          158.64
Median TPOT (ms):                        115.67
P99 TPOT (ms):                           993.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           107.04
Median ITL (ms):                         85.28
P99 ITL (ms):                            569.95
==================================================

优化选项汇总

优化选项 吞吐量提升
引擎选择 +13.0%
量化 +50.6%
启用分块预填充 +7.0%

其他基准测试用例

我们进一步对优化后的配置进行了基准测试，评估其在不同工作负载下的泛化表现。

基准配置的推理服务脚本 
1
vllm serve Qwen/Qwen3-32B
基准配置的测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  1157.25
Total input tokens:                      3200000
Total generated tokens:                  7681
Request throughput (req/s):              0.09
Output token throughput (tok/s):         6.64
Peak output token throughput (tok/s):    24.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          2771.81
---------------Time to First Token----------------
Mean TTFT (ms):                          585981.22
Median TTFT (ms):                        582757.65
P99 TTFT (ms):                           1144686.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          43.42
Median TPOT (ms):                        43.36
P99 TPOT (ms):                           44.34
---------------Inter-token Latency----------------
Mean ITL (ms):                           42.86
Median ITL (ms):                         43.30
P99 ITL (ms):                            46.54
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  976.29
Total input tokens:                      1997942
Total generated tokens:                  99790
Request throughput (req/s):              0.51
Output token throughput (tok/s):         102.21
Peak output token throughput (tok/s):    186.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          2148.69
---------------Time to First Token----------------
Mean TTFT (ms):                          482284.21
Median TTFT (ms):                        483375.28
P99 TTFT (ms):                           956141.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          66.61
Median TPOT (ms):                        62.08
P99 TPOT (ms):                           100.86
---------------Inter-token Latency----------------
Mean ITL (ms):                           66.29
Median ITL (ms):                         43.38
P99 ITL (ms):                            1395.23
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  328.04
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              1.52
Output token throughput (tok/s):         152.42
Peak output token throughput (tok/s):    359.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          3195.30
---------------Time to First Token----------------
Mean TTFT (ms):                          163909.40
Median TTFT (ms):                        164148.38
P99 TTFT (ms):                           322980.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          86.62
Median TPOT (ms):                        84.24
P99 TPOT (ms):                           131.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           85.97
Median ITL (ms):                         44.80
P99 ITL (ms):                            1462.05
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  32.88
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              30.42
Output token throughput (tok/s):         121.66
Peak output token throughput (tok/s):    522.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4005.62
---------------Time to First Token----------------
Mean TTFT (ms):                          21648.67
Median TTFT (ms):                        21342.36
P99 TTFT (ms):                           32678.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1169.72
Median TPOT (ms):                        1292.78
P99 TPOT (ms):                           1417.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           877.29
Median ITL (ms):                         1141.11
P99 ITL (ms):                            1435.80
==================================================
优化后的推理服务脚本 
1
trtllm-serve Qwen/Qwen3-32B-FP8 --max_seq_len=40960 --enable_chunked_prefill
优化后的测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  788.52
Total input tokens:                      3200000
Total generated tokens:                  7832
Request throughput (req/s):              0.13
Output token throughput (tok/s):         9.93
Peak output token throughput (tok/s):    104.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          4068.16
---------------Time to First Token----------------
Mean TTFT (ms):                          396294.90
Median TTFT (ms):                        397717.15
P99 TTFT (ms):                           778540.06
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          286.97
Median TPOT (ms):                        255.80
P99 TPOT (ms):                           581.09
---------------Inter-token Latency----------------
Mean ITL (ms):                           284.98
Median ITL (ms):                         39.76
P99 ITL (ms):                            2105.62
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  425.60
Total input tokens:                      1997942
Total generated tokens:                  99888
Request throughput (req/s):              1.17
Output token throughput (tok/s):         234.70
Peak output token throughput (tok/s):    828.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          4929.16
---------------Time to First Token----------------
Mean TTFT (ms):                          208082.20
Median TTFT (ms):                        204136.20
P99 TTFT (ms):                           415000.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          138.75
Median TPOT (ms):                        141.52
P99 TPOT (ms):                           151.74
---------------Inter-token Latency----------------
Mean ITL (ms):                           138.01
Median ITL (ms):                         45.37
P99 ITL (ms):                            1254.20
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  177.60
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              2.82
Output token throughput (tok/s):         281.53
Peak output token throughput (tok/s):    1529.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          5901.95
---------------Time to First Token----------------
Mean TTFT (ms):                          86596.60
Median TTFT (ms):                        86629.72
P99 TTFT (ms):                           171842.84
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          223.40
Median TPOT (ms):                        237.63
P99 TPOT (ms):                           249.66
---------------Inter-token Latency----------------
Mean ITL (ms):                           221.16
Median ITL (ms):                         49.40
P99 ITL (ms):                            1216.32
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  19.29
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              51.83
Output token throughput (tok/s):         207.33
Peak output token throughput (tok/s):    860.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6826.33
---------------Time to First Token----------------
Mean TTFT (ms):                          10718.84
Median TTFT (ms):                        10511.60
P99 TTFT (ms):                           19055.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          890.11
Median TPOT (ms):                        949.23
P99 TPOT (ms):                           1351.38
---------------Inter-token Latency----------------
Mean ITL (ms):                           667.57
Median ITL (ms):                         925.89
P99 ITL (ms):                            1854.09
==================================================