跳转至

NVIDIA A100 GPUs 上的 Qwen3-32B 吞吐量优化

结论

qwen3-32b-a100

Qwen3-32B 在 A100 GPUs 上的吞吐量优化推荐配置：

推理服务命令 
1
trtllm-serve Qwen/Qwen3-32B --max_seq_len=40960 --enable_chunked_prefill

优化前后的基准测试结果对比：

基准测试用例 基准配置（未做任何优化的 vLLM） 优化后
ShareGPT Total TPS: 1730.98
Mean TPOT(ms): 149.67		 Total TPS: 2054.21 (+18.7%)
Mean TPOT(ms): 89.01
短提示词 Total TPS: 2656.71
Mean TPOT(ms): 528.57		 Total TPS: 3535.54 (+33.1%)
Mean TPOT(ms): 1885.18
中等长度提示词 Total TPS: 2521.38
Mean TPOT(ms): 143.23		 Total TPS: 2715.90 (+7.7%)
Mean TPOT(ms): 145.49
长提示词 Total TPS: 1837.08
Mean TPOT(ms): 96.19		 Total TPS: 2037.67 (+10.9%)
Mean TPOT(ms): 100.09
超长提示词 Total TPS: 1798.45
Mean TPOT(ms): 67.87		 Total TPS: 1580.36 (-13.8%)
Mean TPOT(ms): 47.37

说明

  1. 基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
  2. 其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph、Torch Compile 等。本文结论可作为进一步针对性优化的起点。
  3. 测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

优化目标

在高并发请求场景下实现高吞吐量。

实验环境

模型

Qwen3-32B

硬件

单个 NVIDIA A100 SXM 80GB GPU。

引擎版本

  • vLLM: v0.11.0
  • SGLang: v0.5.5.post1
  • TensorRT-LLM: v1.2.0rc1

基准测试数据集

  1. ShareGPT
  2. 不同序列长度的随机数据集：
    • 超长提示词：32000 输入 tokens，100 输出 tokens
    • 长提示词：4000 输入 tokens，200 输出 tokens
    • 中等长度提示词：2000 输入 tokens，100 输出 tokens
    • 短提示词：128 输入 tokens，4 输出 tokens

基准测试脚本

使用 vLLM bench CLI 工具对模型性能进行基准测试。运行基准测试的命令如下：

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-32B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-32B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

实验结果

1. 选择推理引擎

vLLM

推理服务脚本 
1
vllm serve Qwen/Qwen3-32B --max-model-len 32768
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  241.94
Total input tokens:                      217393
Total generated tokens:                  201396
Request throughput (req/s):              4.13
Output token throughput (tok/s):         832.43
Peak output token throughput (tok/s):    1950.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          1730.98
---------------Time to First Token----------------
Mean TTFT (ms):                          105557.78
Median TTFT (ms):                        105990.93
P99 TTFT (ms):                           209863.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          149.67
Median TPOT (ms):                        123.31
P99 TPOT (ms):                           563.16
---------------Inter-token Latency----------------
Mean ITL (ms):                           117.98
Median ITL (ms):                         61.18
P99 ITL (ms):                            572.34
==================================================

SGLang

推理服务脚本 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-32B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  299.51
Total input tokens:                      217393
Total generated tokens:                  201439
Request throughput (req/s):              3.34
Output token throughput (tok/s):         672.56
Peak output token throughput (tok/s):    1295.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          1398.38
---------------Time to First Token----------------
Mean TTFT (ms):                          139199.11
Median TTFT (ms):                        141086.43
P99 TTFT (ms):                           271684.04
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          75.11
Median TPOT (ms):                        66.68
P99 TPOT (ms):                           149.79
---------------Inter-token Latency----------------
Mean ITL (ms):                           69.65
Median ITL (ms):                         49.10
P99 ITL (ms):                            298.90
==================================================

TensorRT-LLM

推理服务脚本 
1
trtllm-serve Qwen/Qwen3-32B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  203.88
Total input tokens:                      217393
Total generated tokens:                  201412
Request throughput (req/s):              4.90
Output token throughput (tok/s):         987.91
Peak output token throughput (tok/s):    1741.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2054.21
---------------Time to First Token----------------
Mean TTFT (ms):                          84105.84
Median TTFT (ms):                        84356.20
P99 TTFT (ms):                           171829.62
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          89.01
Median TPOT (ms):                        79.85
P99 TPOT (ms):                           265.32
---------------Inter-token Latency----------------
Mean ITL (ms):                           78.87
Median ITL (ms):                         54.75
P99 ITL (ms):                            318.76
==================================================

结果：TensorRT-LLM (2054.21 tok/s) > vLLM (1730.98 tok/s) > SGLang (1398.38 tok/s)

2. TensorRT-LLM/vLLM 中的 量化

TensorRT-LLM FP8

推理服务脚本 
1
trtllm-serve Qwen/Qwen3-32B-FP8
基准测试结果 
1
# RuntimeError: Unsupported SM version for FP8 block scaling GEMM

TensorRT-LLM AWQ

推理服务脚本 
1
trtllm-serve Qwen/Qwen3-32B-AWQ
基准测试结果 
1
# KeyError: 'weight'

vLLM FP8

推理服务脚本 
1
vllm serve Qwen/Qwen3-32B-FP8
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  267.77
Total input tokens:                      217393
Total generated tokens:                  201218
Request throughput (req/s):              3.73
Output token throughput (tok/s):         751.46
Peak output token throughput (tok/s):    1965.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          1563.33
---------------Time to First Token----------------
Mean TTFT (ms):                          103256.52
Median TTFT (ms):                        94937.77
P99 TTFT (ms):                           225680.67
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          339.76
Median TPOT (ms):                        310.91
P99 TPOT (ms):                           944.57
---------------Inter-token Latency----------------
Mean ITL (ms):                           273.16
Median ITL (ms):                         218.10
P99 ITL (ms):                            954.38
==================================================

vLLM AWQ

推理服务脚本 
1
vllm serve Qwen/Qwen3-32B-AWQ
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  237.22
Total input tokens:                      217393
Total generated tokens:                  201832
Request throughput (req/s):              4.22
Output token throughput (tok/s):         850.81
Peak output token throughput (tok/s):    2072.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          1767.21
---------------Time to First Token----------------
Mean TTFT (ms):                          92528.26
Median TTFT (ms):                        85550.21
P99 TTFT (ms):                           202791.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          302.45
Median TPOT (ms):                        277.31
P99 TPOT (ms):                           812.48
---------------Inter-token Latency----------------
Mean ITL (ms):                           243.88
Median ITL (ms):                         203.46
P99 ITL (ms):                            817.18
==================================================

3. TensorRT-LLM 中的 最大批处理 Token 数

推理服务脚本 
1
trtllm-serve Qwen/Qwen3-14B --max_num_tokens=16384
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# --max_num_tokens=16384
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  220.69
Total input tokens:                      217393
Total generated tokens:                  201360
Request throughput (req/s):              4.53
Output token throughput (tok/s):         912.43
Peak output token throughput (tok/s):    1560.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          1897.50
---------------Time to First Token----------------
Mean TTFT (ms):                          90546.36
Median TTFT (ms):                        89254.77
P99 TTFT (ms):                           184658.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          79.75
Median TPOT (ms):                        74.69
P99 TPOT (ms):                           191.75
---------------Inter-token Latency----------------
Mean ITL (ms):                           74.37
Median ITL (ms):                         53.56
P99 ITL (ms):                            325.39
==================================================

# --max_num_tokens=32768
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  257.31
Total input tokens:                      217393
Total generated tokens:                  201414
Request throughput (req/s):              3.89
Output token throughput (tok/s):         782.77
Peak output token throughput (tok/s):    1266.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          1627.65
---------------Time to First Token----------------
Mean TTFT (ms):                          113890.79
Median TTFT (ms):                        114913.27
P99 TTFT (ms):                           225978.59
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          66.22
Median TPOT (ms):                        64.31
P99 TPOT (ms):                           96.69
---------------Inter-token Latency----------------
Mean ITL (ms):                           64.16
Median ITL (ms):                         49.74
P99 ITL (ms):                            258.34
==================================================

优化选项汇总

优化选项 吞吐量提升
引擎选择 +18.7%
量化 -
最大批处理 Token 数 -

其他基准测试用例

我们进一步对优化后的配置进行了基准测试，评估其在不同工作负载下的泛化表现。

基准配置的推理服务脚本 
1
vllm serve Qwen/Qwen3-32B
基准配置的测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  1783.59
Total input tokens:                      3200000
Total generated tokens:                  7699
Request throughput (req/s):              0.06
Output token throughput (tok/s):         4.32
Peak output token throughput (tok/s):    21.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          1798.45
---------------Time to First Token----------------
Mean TTFT (ms):                          890711.38
Median TTFT (ms):                        892708.86
P99 TTFT (ms):                           1760024.36
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          67.87
Median TPOT (ms):                        67.32
P99 TPOT (ms):                           147.19
---------------Inter-token Latency----------------
Mean ITL (ms):                           64.54
Median ITL (ms):                         48.55
P99 ITL (ms):                            690.90
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  1141.94
Total input tokens:                      1997942
Total generated tokens:                  99893
Request throughput (req/s):              0.44
Output token throughput (tok/s):         87.48
Peak output token throughput (tok/s):    180.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          1837.08
---------------Time to First Token----------------
Mean TTFT (ms):                          566579.70
Median TTFT (ms):                        564705.97
P99 TTFT (ms):                           1116945.95
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          96.19
Median TPOT (ms):                        96.40
P99 TPOT (ms):                           99.35
---------------Inter-token Latency----------------
Mean ITL (ms):                           95.82
Median ITL (ms):                         51.55
P99 ITL (ms):                            628.61
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  415.71
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              1.20
Output token throughput (tok/s):         120.27
Peak output token throughput (tok/s):    381.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          2521.38
---------------Time to First Token----------------
Mean TTFT (ms):                          208397.03
Median TTFT (ms):                        209108.79
P99 TTFT (ms):                           408103.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          143.23
Median TPOT (ms):                        143.96
P99 TPOT (ms):                           166.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           142.07
Median ITL (ms):                         51.49
P99 ITL (ms):                            590.04
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  49.57
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              20.17
Output token throughput (tok/s):         80.69
Peak output token throughput (tok/s):    200.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2656.71
---------------Time to First Token----------------
Mean TTFT (ms):                          31177.64
Median TTFT (ms):                        31073.76
P99 TTFT (ms):                           48960.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          528.57
Median TPOT (ms):                        544.55
P99 TPOT (ms):                           547.31
---------------Inter-token Latency----------------
Mean ITL (ms):                           396.42
Median ITL (ms):                         543.22
P99 ITL (ms):                            554.84
==================================================
优化后的推理服务脚本 
1
trtllm-serve Qwen/Qwen3-32B --max_seq_len=40960 --enable_chunked_prefill
优化后的测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  2029.84
Total input tokens:                      3200000
Total generated tokens:                  7880
Request throughput (req/s):              0.05
Output token throughput (tok/s):         3.88
Peak output token throughput (tok/s):    23.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          1580.36
---------------Time to First Token----------------
Mean TTFT (ms):                          1040855.88
Median TTFT (ms):                        1047384.87
P99 TTFT (ms):                           2008642.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          47.37
Median TPOT (ms):                        47.37
P99 TPOT (ms):                           47.49
---------------Inter-token Latency----------------
Mean ITL (ms):                           46.77
Median ITL (ms):                         47.36
P99 ITL (ms):                            49.08
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  1029.52
Total input tokens:                      1997942
Total generated tokens:                  99874
Request throughput (req/s):              0.49
Output token throughput (tok/s):         97.01
Peak output token throughput (tok/s):    241.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          2037.67
---------------Time to First Token----------------
Mean TTFT (ms):                          509693.25
Median TTFT (ms):                        505676.34
P99 TTFT (ms):                           1014720.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          100.09
Median TPOT (ms):                        98.07
P99 TPOT (ms):                           115.57
---------------Inter-token Latency----------------
Mean ITL (ms):                           99.60
Median ITL (ms):                         50.97
P99 ITL (ms):                            2379.87
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  385.94
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              1.30
Output token throughput (tok/s):         129.55
Peak output token throughput (tok/s):    480.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          2715.90
---------------Time to First Token----------------
Mean TTFT (ms):                          189776.20
Median TTFT (ms):                        189321.80
P99 TTFT (ms):                           378730.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          145.49
Median TPOT (ms):                        144.79
P99 TPOT (ms):                           178.50
---------------Inter-token Latency----------------
Mean ITL (ms):                           144.04
Median ITL (ms):                         51.18
P99 ITL (ms):                            2256.49
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  37.25
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              26.85
Output token throughput (tok/s):         107.38
Peak output token throughput (tok/s):    400.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          3535.54
---------------Time to First Token----------------
Mean TTFT (ms):                          19761.77
Median TTFT (ms):                        19497.33
P99 TTFT (ms):                           36592.74
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1885.18
Median TPOT (ms):                        2140.52
P99 TPOT (ms):                           2149.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           1413.87
Median ITL (ms):                         2136.43
P99 ITL (ms):                            2170.23
==================================================