跳转至

昇腾 910B NPUs 上的 Qwen3-8B 吞吐量优化

结论

qwen3-8b-910b

Qwen3-8B 在昇腾 910B NPUs 上的吞吐量优化推荐配置：

推理服务命令 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching

优化前后的基准测试结果对比：

基准测试用例 基准配置（未做任何优化的 vLLM） 优化后
ShareGPT Total TPS: 3143.94
Mean TPOT(ms): 27.61		 Total TPS: 6256.39 (+99.0%)
Mean TPOT(ms): 796.13
短提示词 Total TPS: 5834.79
Mean TPOT(ms): 125.55		 Total TPS: 16611.86 (+184.7%)
Mean TPOT(ms): 1131.26
中等长度提示词 Total TPS: 9589.00
Mean TPOT(ms): 140.39		 Total TPS: 14484.86 (+51.1%)
Mean TPOT(ms): 122.59
长提示词 Total TPS: 7677.41
Mean TPOT(ms): 99.26		 Total TPS: 11159.01 (+45.4%)
Mean TPOT(ms): 97.74
超长提示词 Total TPS: 2890.64
Mean TPOT(ms): 344.18		 Total TPS: 7835.01 (+171.1%)
Mean TPOT(ms): 169.70

说明

  1. 基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
  2. 其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph、Torch Compile 等。本文结论可作为进一步针对性优化的起点。
  3. 测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

优化目标

在高并发请求场景下实现高吞吐量。

实验环境

模型

Qwen/Qwen3-30B-A3B

硬件

单个昇腾 910B NPU。

引擎版本

  • vLLM-Ascend: v0.9.1
  • MindIE: 2.1RC1

基准测试数据集

  1. ShareGPT
  2. 不同序列长度的随机数据集：
    • 超长提示词：32000 输入 tokens，100 输出 tokens
    • 长提示词：4000 输入 tokens，200 输出 tokens
    • 中等长度提示词：2000 输入 tokens，100 输出 tokens
    • 短提示词：128 输入 tokens，4 输出 tokens

基准测试脚本

使用 vLLM bench CLI 工具对模型性能进行基准测试。运行基准测试的命令如下：

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-8B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-8B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

实验结果

1. 选择推理引擎

vLLM

推理服务脚本 
1
vllm serve Qwen/Qwen3-8B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  133.23
Total input tokens:                      217393
Total generated tokens:                  201469
Request throughput (req/s):              7.51
Output token throughput (tok/s):         1512.21
Peak output token throughput (tok/s):    2821.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          3143.94
---------------Time to First Token----------------
Mean TTFT (ms):                          43963.16
Median TTFT (ms):                        41177.19
P99 TTFT (ms):                           91086.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.09
Median TPOT (ms):                        122.07
P99 TPOT (ms):                           269.65
---------------Inter-token Latency----------------
Mean ITL (ms):                           112.90
Median ITL (ms):                         78.96
P99 ITL (ms):                            280.07
==================================================

MindIE

推理服务脚本 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve Qwen/Qwen3-8B
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  74.36
Total input tokens:                      217393
Total generated tokens:                  201609
Request throughput (req/s):              13.45
Output token throughput (tok/s):         2711.10
Peak output token throughput (tok/s):    4588.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5634.45
---------------Time to First Token----------------
Mean TTFT (ms):                          8242.22
Median TTFT (ms):                        8298.00
P99 TTFT (ms):                           18074.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          788.31
Median TPOT (ms):                        204.41
P99 TPOT (ms):                           6547.97
---------------Inter-token Latency----------------
Mean ITL (ms):                           155.96
Median ITL (ms):                         45.35
P99 ITL (ms):                            218.57
==================================================

结果：MineIE (5634.4 tok/s) > vLLM (3143.94 tok/s)

2. MindIE 中的 量化

推理服务脚本 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  69.32
Total input tokens:                      217393
Total generated tokens:                  201315
Request throughput (req/s):              14.42
Output token throughput (tok/s):         2903.93
Peak output token throughput (tok/s):    4799.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6039.79
---------------Time to First Token----------------
Mean TTFT (ms):                          5834.61
Median TTFT (ms):                        5648.87
P99 TTFT (ms):                           11181.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          822.87
Median TPOT (ms):                        188.93
P99 TPOT (ms):                           7710.20
---------------Inter-token Latency----------------
Mean ITL (ms):                           148.97
Median ITL (ms):                         45.31
P99 ITL (ms):                            97.87
==================================================

3. 在 MindIE 中启用前缀缓存

推理服务脚本 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  66.96
Total input tokens:                      217393
Total generated tokens:                  201517
Request throughput (req/s):              14.93
Output token throughput (tok/s):         3009.64
Peak output token throughput (tok/s):    4985.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6256.39
---------------Time to First Token----------------
Mean TTFT (ms):                          5320.32
Median TTFT (ms):                        5159.10
P99 TTFT (ms):                           10384.83
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          796.13
Median TPOT (ms):                        183.35
P99 TPOT (ms):                           7267.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           141.91
Median ITL (ms):                         43.55
P99 ITL (ms):                            115.20
==================================================

4. MindIE 中的 批大小调优

推理服务脚本 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching --max-batch-size=400 --max-prefill-batch-size=200
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  69.07
Total input tokens:                      217393
Total generated tokens:                  201424
Request throughput (req/s):              14.48
Output token throughput (tok/s):         2916.34
Peak output token throughput (tok/s):    6017.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6063.89
---------------Time to First Token----------------
Mean TTFT (ms):                          6335.34
Median TTFT (ms):                        5812.45
P99 TTFT (ms):                           11260.65
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          581.93
Median TPOT (ms):                        167.14
P99 TPOT (ms):                           5501.66
---------------Inter-token Latency----------------
Mean ITL (ms):                           143.92
Median ITL (ms):                         70.27
P99 ITL (ms):                            281.16
==================================================

优化选项汇总

优化选项 吞吐量提升
引擎选择 +79.2%
量化 +7.2%
前缀缓存 +3.6%
批大小调优 -

其他基准测试用例

我们进一步对优化后的配置进行了基准测试，评估其在不同工作负载下的泛化表现。

基准配置的推理服务脚本 
1
vllm serve Qwen/Qwen3-8B
基准配置的测试结果 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  1110.48
Total input tokens:                      3200000
Total generated tokens:                  10000
Request throughput (req/s):              0.09
Output token throughput (tok/s):         9.01
Peak output token throughput (tok/s):    147.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          2890.64
---------------Time to First Token----------------
Mean TTFT (ms):                          559550.01
Median TTFT (ms):                        561560.86
P99 TTFT (ms):                           1096851.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          344.18
Median TPOT (ms):                        346.89
P99 TPOT (ms):                           668.43
---------------Inter-token Latency----------------
Mean ITL (ms):                           340.74
Median ITL (ms):                         48.54
P99 ITL (ms):                            59.04
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  273.17
Total input tokens:                      1997463
Total generated tokens:                  99812
Request throughput (req/s):              1.83
Output token throughput (tok/s):         365.38
Peak output token throughput (tok/s):    1062.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          7677.41
---------------Time to First Token----------------
Mean TTFT (ms):                          132541.20
Median TTFT (ms):                        135284.67
P99 TTFT (ms):                           262452.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          99.26
Median TPOT (ms):                        100.01
P99 TPOT (ms):                           175.24
---------------Inter-token Latency----------------
Mean ITL (ms):                           98.69
Median ITL (ms):                         55.84
P99 ITL (ms):                            134.52
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  109.30
Total input tokens:                      998057
Total generated tokens:                  50000
Request throughput (req/s):              4.57
Output token throughput (tok/s):         457.47
Peak output token throughput (tok/s):    1904.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          9589.00
---------------Time to First Token----------------
Mean TTFT (ms):                          53978.52
Median TTFT (ms):                        57661.39
P99 TTFT (ms):                           103428.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          140.39
Median TPOT (ms):                        142.80
P99 TPOT (ms):                           275.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           139.05
Median ITL (ms):                         63.53
P99 ITL (ms):                            627.10
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  22.59
Total input tokens:                      127782
Total generated tokens:                  4000
Request throughput (req/s):              44.28
Output token throughput (tok/s):         177.10
Peak output token throughput (tok/s):    1280.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5834.79
---------------Time to First Token----------------
Mean TTFT (ms):                          18464.39
Median TTFT (ms):                        17489.55
P99 TTFT (ms):                           22134.55
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.55
Median TPOT (ms):                        135.76
P99 TPOT (ms):                           149.27
---------------Inter-token Latency----------------
Mean ITL (ms):                           94.16
Median ITL (ms):                         83.83
P99 ITL (ms):                            251.91
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  275.57
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              0.36
Output token throughput (tok/s):         80.06
Peak output token throughput (tok/s):    92.00
Peak concurrent requests:                8.00
Total Token throughput (tok/s):          164.46
---------------Time to First Token----------------
Mean TTFT (ms):                          145.88
Median TTFT (ms):                        142.78
P99 TTFT (ms):                           201.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          47.28
Median TPOT (ms):                        47.29
P99 TPOT (ms):                           50.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           47.10
Median ITL (ms):                         46.72
P99 ITL (ms):                            91.82
==================================================
优化后的推理服务脚本 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching
优化后的测试结果 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  296.36
Total input tokens:                      3200000
Total generated tokens:                  9693
Request throughput (req/s):              0.34
Output token throughput (tok/s):         32.71
Peak output token throughput (tok/s):    330.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          10830.52
---------------Time to First Token----------------
Mean TTFT (ms):                          147133.58
Median TTFT (ms):                        147851.57
P99 TTFT (ms):                           289974.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          242.08
Median TPOT (ms):                        232.74
P99 TPOT (ms):                           741.49
---------------Inter-token Latency----------------
Mean ITL (ms):                           225.71
Median ITL (ms):                         46.46
P99 ITL (ms):                            56.81
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  157.15
Total input tokens:                      2000000
Total generated tokens:                  99961
Request throughput (req/s):              3.18
Output token throughput (tok/s):         636.07
Peak output token throughput (tok/s):    2142.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          13362.49
---------------Time to First Token----------------
Mean TTFT (ms):                          70716.71
Median TTFT (ms):                        74982.88
P99 TTFT (ms):                           148427.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          130.97
Median TPOT (ms):                        134.98
P99 TPOT (ms):                           208.16
---------------Inter-token Latency----------------
Mean ITL (ms):                           130.95
Median ITL (ms):                         63.84
P99 ITL (ms):                            489.52
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  67.34
Total input tokens:                      1000000
Total generated tokens:                  49975
Request throughput (req/s):              7.42
Output token throughput (tok/s):         742.11
Peak output token throughput (tok/s):    3000.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          15591.71
---------------Time to First Token----------------
Mean TTFT (ms):                          29078.68
Median TTFT (ms):                        32203.66
P99 TTFT (ms):                           62077.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          194.54
Median TPOT (ms):                        192.07
P99 TPOT (ms):                           358.38
---------------Inter-token Latency----------------
Mean ITL (ms):                           194.46
Median ITL (ms):                         68.54
P99 ITL (ms):                            4496.71
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  8.78
Total input tokens:                      128000
Total generated tokens:                  4000
Request throughput (req/s):              113.94
Output token throughput (tok/s):         455.76
Peak output token throughput (tok/s):    2511.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          15040.15
---------------Time to First Token----------------
Mean TTFT (ms):                          4491.88
Median TTFT (ms):                        4467.47
P99 TTFT (ms):                           7693.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1249.72
Median TPOT (ms):                        1258.92
P99 TPOT (ms):                           2252.88
---------------Inter-token Latency----------------
Mean ITL (ms):                           1249.72
Median ITL (ms):                         54.67
P99 ITL (ms):                            6378.56
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             4
Benchmark duration (s):                  1292.33
Total input tokens:                      217393
Total generated tokens:                  201773
Request throughput (req/s):              0.77
Output token throughput (tok/s):         156.13
Peak output token throughput (tok/s):    200.00
Peak concurrent requests:                9.00
Total Token throughput (tok/s):          324.35
---------------Time to First Token----------------
Mean TTFT (ms):                          112.38
Median TTFT (ms):                        111.05
P99 TTFT (ms):                           154.06
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.42
Median TPOT (ms):                        24.47
P99 TPOT (ms):                           35.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.08
Median ITL (ms):                         23.05
P99 ITL (ms):                            79.54
==================================================