跳转至

NVIDIA H100 GPUs 上的 Qwen3-14B 吞吐量优化

结论

qwen3-14b-h100

Qwen3-14B 在 H100 GPUs 上的吞吐量优化推荐配置：

推理服务命令 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-14B-FP8

优化前后的基准测试结果对比：

基准测试用例 基准配置（未做任何优化的 vLLM） 优化后
ShareGPT Total TPS: 6595.66
Mean TPOT(ms): 226.03		 Total TPS: 10426.27 (+58.1%)
Mean TPOT(ms): 251.67
短提示词 Total TPS: 5796.73
Mean TPOT(ms): 525.31		 Total TPS: 16884.75 (+191.3%)
Mean TPOT(ms): 927.88
中等长度提示词 Total TPS: 10597.19
Mean TPOT(ms): 206.51		 Total TPS: 14764.26 (+39.3%)
Mean TPOT(ms): 143.22
长提示词 Total TPS: 9218.03
Mean TPOT(ms): 132.45		 Total TPS: 12257.62 (+33.0%)
Mean TPOT(ms): 84.93
超长提示词 Total TPS: 8298.56
Mean TPOT(ms): 277.44		 Total TPS: 10014.29 (+20.7%)
Mean TPOT(ms): 215.93

说明

  1. 基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
  2. 其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph、Torch Compile 等。本文结论可作为进一步针对性优化的起点。
  3. 测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。
  4. 量化可能影响准确率，但对于大多数模型，FP8 量化的准确率下降可控制在 1% 以内。详情参见评估结果。因此，强烈建议在高吞吐量推理服务场景中使用 FP8 量化。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

优化目标

在高并发请求场景下实现高吞吐量。

实验环境

模型

Qwen3-14B

硬件

单个 NVIDIA H100 SXM GPU。

引擎版本

  • vLLM: v0.11.0
  • SGLang: v0.5.5.post1
  • TensorRT-LLM: v1.2.0rc1

基准测试数据集

  1. ShareGPT
  2. 不同序列长度的随机数据集：
    • 超长提示词：32000 输入 tokens，100 输出 tokens
    • 长提示词：4000 输入 tokens，200 输出 tokens
    • 中等长度提示词：2000 输入 tokens，100 输出 tokens
    • 短提示词：128 输入 tokens，4 输出 tokens

基准测试脚本

使用 vLLM bench CLI 工具对模型性能进行基准测试。运行基准测试的命令如下：

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-14B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-14B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

实验结果

1. 选择推理引擎

vLLM

推理服务脚本 
1
vllm serve Qwen/Qwen3-14B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  63.53
Total input tokens:                      217393
Total generated tokens:                  201648
Request throughput (req/s):              15.74
Output token throughput (tok/s):         3173.92
Peak output token throughput (tok/s):    8890.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6595.66
---------------Time to First Token----------------
Mean TTFT (ms):                          22141.21
Median TTFT (ms):                        21872.31
P99 TTFT (ms):                           31156.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          226.03
Median TPOT (ms):                        117.46
P99 TPOT (ms):                           621.87
---------------Inter-token Latency----------------
Mean ITL (ms):                           92.83
Median ITL (ms):                         59.99
P99 ITL (ms):                            627.72
==================================================

SGLang

推理服务脚本 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-14B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  49.86
Total input tokens:                      217393
Total generated tokens:                  201628
Request throughput (req/s):              20.06
Output token throughput (tok/s):         4043.89
Peak output token throughput (tok/s):    12059.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          8403.97
---------------Time to First Token----------------
Mean TTFT (ms):                          9230.85
Median TTFT (ms):                        8812.15
P99 TTFT (ms):                           19581.07
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          235.33
Median TPOT (ms):                        118.67
P99 TPOT (ms):                           1462.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           91.54
Median ITL (ms):                         51.39
P99 ITL (ms):                            442.65
==================================================

TensorRT-LLM

推理服务脚本 
1
trtllm-serve Qwen/Qwen3-14B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  50.39
Total input tokens:                      217393
Total generated tokens:                  201612
Request throughput (req/s):              19.84
Output token throughput (tok/s):         4000.99
Peak output token throughput (tok/s):    8281.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          8315.15
---------------Time to First Token----------------
Mean TTFT (ms):                          10230.91
Median TTFT (ms):                        9088.74
P99 TTFT (ms):                           24363.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          161.58
Median TPOT (ms):                        107.16
P99 TPOT (ms):                           579.49
---------------Inter-token Latency----------------
Mean ITL (ms):                           84.78
Median ITL (ms):                         0.12
P99 ITL (ms):                            1440.77
==================================================

结果：SGLang (8424.86 tok/s) > TensorRT-LLM (8315.15 tok/s) > vLLM (6595.66 tok/s)

2. SGLang 中的 量化

FP8

推理服务脚本 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-14B-FP8
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  40.21
Total input tokens:                      217393
Total generated tokens:                  201833
Request throughput (req/s):              24.87
Output token throughput (tok/s):         5019.64
Peak output token throughput (tok/s):    14199.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10426.27
---------------Time to First Token----------------
Mean TTFT (ms):                          7320.60
Median TTFT (ms):                        7110.49
P99 TTFT (ms):                           13187.08
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          251.67
Median TPOT (ms):                        94.53
P99 TPOT (ms):                           1678.80
---------------Inter-token Latency----------------
Mean ITL (ms):                           77.06
Median ITL (ms):                         48.45
P99 ITL (ms):                            383.61
==================================================

3. SGLang 中的 注意力后端

推理服务脚本 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-14B-FP8 --attention-backend=fa3
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
# fa3
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  40.18
Total input tokens:                      217393
Total generated tokens:                  201833
Request throughput (req/s):              24.89
Output token throughput (tok/s):         5023.58
Peak output token throughput (tok/s):    13005.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10434.44
---------------Time to First Token----------------
Mean TTFT (ms):                          7304.47
Median TTFT (ms):                        7081.19
P99 TTFT (ms):                           13629.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          242.72
Median TPOT (ms):                        94.43
P99 TPOT (ms):                           1559.90
---------------Inter-token Latency----------------
Mean ITL (ms):                           76.97
Median ITL (ms):                         47.55
P99 ITL (ms):                            359.57
==================================================

# --attention-backend=flashinfer
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  45.91
Total input tokens:                      217393
Total generated tokens:                  201462
Request throughput (req/s):              21.78
Output token throughput (tok/s):         4388.17
Peak output token throughput (tok/s):    15814.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          9123.34
---------------Time to First Token----------------
Mean TTFT (ms):                          13440.31
Median TTFT (ms):                        13197.52
P99 TTFT (ms):                           19982.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          246.57
Median TPOT (ms):                        95.09
P99 TPOT (ms):                           1613.59
---------------Inter-token Latency----------------
Mean ITL (ms):                           76.95
Median ITL (ms):                         48.29
P99 ITL (ms):                            367.15
==================================================

# --attention-backend=triton
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  41.67
Total input tokens:                      217393
Total generated tokens:                  201602
Request throughput (req/s):              24.00
Output token throughput (tok/s):         4838.09
Peak output token throughput (tok/s):    14024.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10055.13
---------------Time to First Token----------------
Mean TTFT (ms):                          7297.40
Median TTFT (ms):                        6973.27
P99 TTFT (ms):                           14354.40
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          268.09
Median TPOT (ms):                        100.54
P99 TPOT (ms):                           1810.20
---------------Inter-token Latency----------------
Mean ITL (ms):                           82.50
Median ITL (ms):                         51.79
P99 ITL (ms):                            379.98
==================================================

优化选项汇总

优化选项 吞吐量提升
引擎选择 +27.7%
量化 +23.8%
注意力后端 -

其他基准测试用例

我们进一步对优化后的配置进行了基准测试，评估其在不同工作负载下的泛化表现。

基准配置的推理服务脚本 
1
vllm serve Qwen/Qwen3-14B
基准配置的测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  386.80
Total input tokens:                      3200000
Total generated tokens:                  9855
Request throughput (req/s):              0.26
Output token throughput (tok/s):         25.48
Peak output token throughput (tok/s):    192.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          8298.56
---------------Time to First Token----------------
Mean TTFT (ms):                          191024.43
Median TTFT (ms):                        191051.49
P99 TTFT (ms):                           380407.97
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          277.44
Median TPOT (ms):                        284.52
P99 TPOT (ms):                           324.14
---------------Inter-token Latency----------------
Mean ITL (ms):                           274.38
Median ITL (ms):                         43.48
P99 ITL (ms):                            1101.18
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  227.57
Total input tokens:                      1997942
Total generated tokens:                  99803
Request throughput (req/s):              2.20
Output token throughput (tok/s):         438.56
Peak output token throughput (tok/s):    1536.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          9218.03
---------------Time to First Token----------------
Mean TTFT (ms):                          112716.92
Median TTFT (ms):                        110734.54
P99 TTFT (ms):                           219749.74
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          132.45
Median TPOT (ms):                        139.68
P99 TPOT (ms):                           161.93
---------------Inter-token Latency----------------
Mean ITL (ms):                           132.11
Median ITL (ms):                         43.93
P99 ITL (ms):                            656.79
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  98.91
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              5.06
Output token throughput (tok/s):         505.51
Peak output token throughput (tok/s):    2890.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          10597.19
---------------Time to First Token----------------
Mean TTFT (ms):                          50716.13
Median TTFT (ms):                        49138.28
P99 TTFT (ms):                           94614.51
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          206.51
Median TPOT (ms):                        231.62
P99 TPOT (ms):                           236.67
---------------Inter-token Latency----------------
Mean ITL (ms):                           204.84
Median ITL (ms):                         46.01
P99 ITL (ms):                            642.06
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  22.72
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              44.02
Output token throughput (tok/s):         176.06
Peak output token throughput (tok/s):    843.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5796.73
---------------Time to First Token----------------
Mean TTFT (ms):                          18282.75
Median TTFT (ms):                        18061.13
P99 TTFT (ms):                           22564.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          525.31
Median TPOT (ms):                        597.29
P99 TPOT (ms):                           610.81
---------------Inter-token Latency----------------
Mean ITL (ms):                           393.98
Median ITL (ms):                         581.28
P99 ITL (ms):                            633.07
==================================================
优化后的推理服务脚本 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-14B-FP8
优化后的测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  320.54
Total input tokens:                      3200000
Total generated tokens:                  9964
Request throughput (req/s):              0.31
Output token throughput (tok/s):         31.09
Peak output token throughput (tok/s):    250.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          10014.29
---------------Time to First Token----------------
Mean TTFT (ms):                          158068.68
Median TTFT (ms):                        157591.30
P99 TTFT (ms):                           313702.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          215.93
Median TPOT (ms):                        212.59
P99 TPOT (ms):                           326.36
---------------Inter-token Latency----------------
Mean ITL (ms):                           211.53
Median ITL (ms):                         41.38
P99 ITL (ms):                            5675.37
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  171.14
Total input tokens:                      1997942
Total generated tokens:                  99814
Request throughput (req/s):              2.92
Output token throughput (tok/s):         583.23
Peak output token throughput (tok/s):    1950.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          12257.62
---------------Time to First Token----------------
Mean TTFT (ms):                          82073.47
Median TTFT (ms):                        85092.81
P99 TTFT (ms):                           165401.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          84.93
Median TPOT (ms):                        83.43
P99 TPOT (ms):                           131.23
---------------Inter-token Latency----------------
Mean ITL (ms):                           84.30
Median ITL (ms):                         41.87
P99 ITL (ms):                            89.35
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  70.99
Total input tokens:                      998175
Total generated tokens:                  50000
Request throughput (req/s):              7.04
Output token throughput (tok/s):         704.28
Peak output token throughput (tok/s):    3496.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          14764.26
---------------Time to First Token----------------
Mean TTFT (ms):                          33245.08
Median TTFT (ms):                        33169.68
P99 TTFT (ms):                           68799.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          143.22
Median TPOT (ms):                        146.25
P99 TPOT (ms):                           216.28
---------------Inter-token Latency----------------
Mean ITL (ms):                           140.52
Median ITL (ms):                         45.98
P99 ITL (ms):                            1987.27
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  7.80
Total input tokens:                      127698
Total generated tokens:                  4000
Request throughput (req/s):              128.21
Output token throughput (tok/s):         512.83
Peak output token throughput (tok/s):    3784.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          16884.75
---------------Time to First Token----------------
Mean TTFT (ms):                          4489.43
Median TTFT (ms):                        4057.99
P99 TTFT (ms):                           7239.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          927.88
Median TPOT (ms):                        962.87
P99 TPOT (ms):                           1793.61
---------------Inter-token Latency----------------
Mean ITL (ms):                           556.73
Median ITL (ms):                         82.38
P99 ITL (ms):                            5043.60
==================================================