跳转至

昇腾 910B NPUs 上的 Qwen3-30B-A3B 吞吐量优化

结论

qwen3-30b-a3b-910b

Qwen3-30B-A3B 在昇腾 910B NPUs 上的吞吐量优化推荐配置：

推理服务命令 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve Qwen/Qwen3-30B-A3B

优化前后的基准测试结果对比：

基准测试用例 基准配置（未做任何优化的 vLLM） 优化后
ShareGPT Total TPS: 1227.19
Mean TPOT(ms): 318.03		 Total TPS: 4712.47 (+284.0%)
Mean TPOT(ms): 964.45
短提示词 Total TPS: 5340.68
Mean TPOT(ms): 312.49		 Total TPS: 15040.15 (+181.6%)
Mean TPOT(ms): 1249.72
中等长度提示词 Total TPS: 7843.85
Mean TPOT(ms): 344.86		 Total TPS: 15591.71 (+98.8%)
Mean TPOT(ms): 194.54
长提示词 Total TPS: 5881.36
Mean TPOT(ms): 277.5		 Total TPS: 13362.49 (+127.2%)
Mean TPOT(ms): 130.97
超长提示词 Total TPS: 3704.57
Mean TPOT(ms): 503.84		 Total TPS: 10830.52 (+192.4%)
Mean TPOT(ms): 242.08

说明

  1. 基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
  2. 其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph、Torch Compile 等。本文结论可作为进一步针对性优化的起点。
  3. 测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

优化目标

在高并发请求场景下实现高吞吐量。

实验环境

模型

Qwen/Qwen3-30B-A3B

硬件

单个昇腾 910B NPU。

引擎版本

  • vLLM-Ascend: v0.9.1
  • MindIE: 2.1RC1

基准测试数据集

  1. ShareGPT
  2. 不同序列长度的随机数据集：
    • 超长提示词：32000 输入 tokens，100 输出 tokens
    • 长提示词：4000 输入 tokens，200 输出 tokens
    • 中等长度提示词：2000 输入 tokens，100 输出 tokens
    • 短提示词：128 输入 tokens，4 输出 tokens

基准测试脚本

使用 vLLM bench CLI 工具对模型性能进行基准测试。运行基准测试的命令如下：

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-30B-A3B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-30B-A3B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

实验结果

1. 选择推理引擎

vLLM

推理服务脚本 
1
vllm serve Qwen/Qwen3-30B-A3B -tp 2
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  341.49
Total input tokens:                      217393
Total generated tokens:                  201675
Request throughput (req/s):              2.93
Output token throughput (tok/s):         590.58
Peak output token throughput (tok/s):    1283.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          1227.19
---------------Time to First Token----------------
Mean TTFT (ms):                          91791.47
Median TTFT (ms):                        85430.73
P99 TTFT (ms):                           217017.19
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          318.03
Median TPOT (ms):                        326.12
P99 TPOT (ms):                           376.59
---------------Inter-token Latency----------------
Mean ITL (ms):                           302.02
Median ITL (ms):                         245.14
P99 ITL (ms):                            557.35
==================================================

MindIE

推理服务脚本 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve Qwen3-30B-A3B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  88.95
Total input tokens:                      217393
Total generated tokens:                  201779
Request throughput (req/s):              11.24
Output token throughput (tok/s):         2268.46
Peak output token throughput (tok/s):    3394.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4712.47
---------------Time to First Token----------------
Mean TTFT (ms):                          6315.01
Median TTFT (ms):                        6421.94
P99 TTFT (ms):                           11305.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          964.45
Median TPOT (ms):                        242.38
P99 TPOT (ms):                           8546.71
---------------Inter-token Latency----------------
Mean ITL (ms):                           183.79
Median ITL (ms):                         60.42
P99 ITL (ms):                            74.07
==================================================

结果：MineIE (4712.47 tok/s) > vLLM (1227.19 tok/s)

2. 在 MindIE 中启用前缀缓存

推理服务脚本 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve Qwen3-30B-A3B --enable-prefix-caching
基准测试结果 
1
# Crash, not supported for the model

3. MindIE 中的 量化

推理服务脚本 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  92.29
Total input tokens:                      217393
Total generated tokens:                  201586
Request throughput (req/s):              10.84
Output token throughput (tok/s):         2184.30
Peak output token throughput (tok/s):    3392.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4539.88
---------------Time to First Token----------------
Mean TTFT (ms):                          5588.80
Median TTFT (ms):                        4708.65
P99 TTFT (ms):                           18446.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          988.22
Median TPOT (ms):                        240.48
P99 TPOT (ms):                           11241.85
---------------Inter-token Latency----------------
Mean ITL (ms):                           195.17
Median ITL (ms):                         62.16
P99 ITL (ms):                            226.37
==================================================

优化选项汇总

优化选项 吞吐量提升
引擎选择 +284.0%
前缀缓存 -
量化 -

其他基准测试用例

我们进一步对优化后的配置进行了基准测试，评估其在不同工作负载下的泛化表现。

基准配置的推理服务脚本 
1
vllm serve Qwen/Qwen3-30B-A3B -tp 2
基准配置的测试结果 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  866.50
Total input tokens:                      3200000
Total generated tokens:                  10000
Request throughput (req/s):              0.12
Output token throughput (tok/s):         11.54
Peak output token throughput (tok/s):    117.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          3704.57
---------------Time to First Token----------------
Mean TTFT (ms):                          442404.81
Median TTFT (ms):                        440136.40
P99 TTFT (ms):                           853640.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          503.84
Median TPOT (ms):                        498.47
P99 TPOT (ms):                           998.84
---------------Inter-token Latency----------------
Mean ITL (ms):                           498.80
Median ITL (ms):                         125.10
P99 ITL (ms):                            177.10
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  357.03
Total input tokens:                      2000000
Total generated tokens:                  99842
Request throughput (req/s):              1.40
Output token throughput (tok/s):         279.64
Peak output token throughput (tok/s):    864.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          5881.36
---------------Time to First Token----------------
Mean TTFT (ms):                          156755.15
Median TTFT (ms):                        163311.37
P99 TTFT (ms):                           314736.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          277.51
Median TPOT (ms):                        273.18
P99 TPOT (ms):                           452.09
---------------Inter-token Latency----------------
Mean ITL (ms):                           276.20
Median ITL (ms):                         243.71
P99 ITL (ms):                            499.20
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  133.86
Total input tokens:                      1000000
Total generated tokens:                  50000
Request throughput (req/s):              3.74
Output token throughput (tok/s):         373.52
Peak output token throughput (tok/s):    1080.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          7843.85
---------------Time to First Token----------------
Mean TTFT (ms):                          56434.48
Median TTFT (ms):                        62622.42
P99 TTFT (ms):                           117368.72
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          344.86
Median TPOT (ms):                        350.15
P99 TPOT (ms):                           667.15
---------------Inter-token Latency----------------
Mean ITL (ms):                           341.55
Median ITL (ms):                         240.26
P99 ITL (ms):                            1189.67
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  24.72
Total input tokens:                      128000
Total generated tokens:                  4000
Request throughput (req/s):              40.46
Output token throughput (tok/s):         161.84
Peak output token throughput (tok/s):    768.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5340.68
---------------Time to First Token----------------
Mean TTFT (ms):                          19130.06
Median TTFT (ms):                        17860.56
P99 TTFT (ms):                           23751.62
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          312.49
Median TPOT (ms):                        308.02
P99 TPOT (ms):                           338.42
---------------Inter-token Latency----------------
Mean ITL (ms):                           234.36
Median ITL (ms):                         269.04
P99 ITL (ms):                            465.28
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             4
Benchmark duration (s):                  6271.85
Total input tokens:                      217393
Total generated tokens:                  201847
Request throughput (req/s):              0.16
Output token throughput (tok/s):         32.18
Peak output token throughput (tok/s):    40.00
Peak concurrent requests:                7.00
Total Token throughput (tok/s):          66.84
---------------Time to First Token----------------
Mean TTFT (ms):                          359.99
Median TTFT (ms):                        357.20
P99 TTFT (ms):                           445.74
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          122.71
Median TPOT (ms):                        122.70
P99 TPOT (ms):                           134.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           122.00
Median ITL (ms):                         120.56
P99 ITL (ms):                            236.94
==================================================
优化后的推理服务脚本 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve Qwen3-30B-A3B
优化后的测试结果 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  296.36
Total input tokens:                      3200000
Total generated tokens:                  9693
Request throughput (req/s):              0.34
Output token throughput (tok/s):         32.71
Peak output token throughput (tok/s):    330.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          10830.52
---------------Time to First Token----------------
Mean TTFT (ms):                          147133.58
Median TTFT (ms):                        147851.57
P99 TTFT (ms):                           289974.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          242.08
Median TPOT (ms):                        232.74
P99 TPOT (ms):                           741.49
---------------Inter-token Latency----------------
Mean ITL (ms):                           225.71
Median ITL (ms):                         46.46
P99 ITL (ms):                            56.81
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  157.15
Total input tokens:                      2000000
Total generated tokens:                  99961
Request throughput (req/s):              3.18
Output token throughput (tok/s):         636.07
Peak output token throughput (tok/s):    2142.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          13362.49
---------------Time to First Token----------------
Mean TTFT (ms):                          70716.71
Median TTFT (ms):                        74982.88
P99 TTFT (ms):                           148427.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          130.97
Median TPOT (ms):                        134.98
P99 TPOT (ms):                           208.16
---------------Inter-token Latency----------------
Mean ITL (ms):                           130.95
Median ITL (ms):                         63.84
P99 ITL (ms):                            489.52
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  67.34
Total input tokens:                      1000000
Total generated tokens:                  49975
Request throughput (req/s):              7.42
Output token throughput (tok/s):         742.11
Peak output token throughput (tok/s):    3000.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          15591.71
---------------Time to First Token----------------
Mean TTFT (ms):                          29078.68
Median TTFT (ms):                        32203.66
P99 TTFT (ms):                           62077.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          194.54
Median TPOT (ms):                        192.07
P99 TPOT (ms):                           358.38
---------------Inter-token Latency----------------
Mean ITL (ms):                           194.46
Median ITL (ms):                         68.54
P99 ITL (ms):                            4496.71
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  8.78
Total input tokens:                      128000
Total generated tokens:                  4000
Request throughput (req/s):              113.94
Output token throughput (tok/s):         455.76
Peak output token throughput (tok/s):    2511.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          15040.15
---------------Time to First Token----------------
Mean TTFT (ms):                          4491.88
Median TTFT (ms):                        4467.47
P99 TTFT (ms):                           7693.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1249.72
Median TPOT (ms):                        1258.92
P99 TPOT (ms):                           2252.88
---------------Inter-token Latency----------------
Mean ITL (ms):                           1249.72
Median ITL (ms):                         54.67
P99 ITL (ms):                            6378.56
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             4
Benchmark duration (s):                  1292.33
Total input tokens:                      217393
Total generated tokens:                  201773
Request throughput (req/s):              0.77
Output token throughput (tok/s):         156.13
Peak output token throughput (tok/s):    200.00
Peak concurrent requests:                9.00
Total Token throughput (tok/s):          324.35
---------------Time to First Token----------------
Mean TTFT (ms):                          112.38
Median TTFT (ms):                        111.05
P99 TTFT (ms):                           154.06
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.42
Median TPOT (ms):                        24.47
P99 TPOT (ms):                           35.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.08
Median ITL (ms):                         23.05
P99 ITL (ms):                            79.54
==================================================