コンテンツにスキップ

ASCEND 910B NPU 上での Qwen3-8B のスループット最適化

結論

qwen3-8b-910b

ASCEND 910B NPU 上で Qwen3-8B のスループットを最適化するための推奨設定：

推論サービスのコマンド 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching

最適化前後のベンチマーク結果の比較：

ベンチマークケース 基準設定（最適化なしの vLLM） 最適化後
ShareGPT 合計 TPS: 3143.94
平均 TPOT(ms): 27.61		 合計 TPS: 6256.39 (+99.0%)
平均 TPOT(ms): 796.13
短いプロンプト 合計 TPS: 5834.79
平均 TPOT(ms): 125.55		 合計 TPS: 16611.86 (+184.7%)
平均 TPOT(ms): 1131.26
中程度のプロンプト 合計 TPS: 9589.00
平均 TPOT(ms): 140.39		 合計 TPS: 14484.86 (+51.1%)
平均 TPOT(ms): 122.59
長いプロンプト 合計 TPS: 7677.41
平均 TPOT(ms): 99.26		 合計 TPS: 11159.01 (+45.4%)
平均 TPOT(ms): 97.74
非常に長いプロンプト 合計 TPS: 2890.64
平均 TPOT(ms): 344.18		 合計 TPS: 7835.01 (+171.1%)
平均 TPOT(ms): 169.70

注記

  1. 本ベンチマークは、考えられるすべての最適化の組み合わせを網羅していません。たとえば、デフォルト設定で最も性能が高い推論エンジンを選び、それを追加調整の出発点としています。このような候補の絞り込みで得られるのは局所最適解であり、大域最適解とは限りません。
  2. 最大バッチサイズ、スケジューリング設定、拡張 KV キャッシュ、CUDA graph、Torch Compile など、利用シナリオに応じた最適化もあります。本書の結論は、用途に合わせてさらに最適化するための出発点として利用できます。
  3. 本テストは特定のハードウェアとソフトウェアの構成で実施しています。推論エンジンの進化により、結論が変わる可能性があります。

記載漏れや最新の変更を反映すべき点があれば、お知らせください。

最適化の目標

高並列度のリクエストで高スループットを実現します。

実験環境

モデル

Qwen/Qwen3-30B-A3B

ハードウェア

単一の Ascend 910B NPU。

エンジンのバージョン

  • vLLM-Ascend: v0.9.1
  • MindIE: 2.1RC1

ベンチマークデータセット

  1. ShareGPT
  2. シーケンス長を変えたランダムデータセット：
    • 非常に長いプロンプト： 32000 入力トークン, 100 出力トークン
    • 長いプロンプト： 4000 入力トークン, 200 出力トークン
    • 中程度のプロンプト： 2000 入力トークン, 100 出力トークン
    • 短いプロンプト： 128 入力トークン, 4 出力トークン

ベンチマークスクリプト

モデルの性能評価には vLLM bench CLI ツールを使用します。ベンチマークを実行するコマンドは次のとおりです：

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-8B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-8B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

実験結果

1. 推論エンジンの選択

vLLM

推論サービスのスクリプト 
1
vllm serve Qwen/Qwen3-8B
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  133.23
Total input tokens:                      217393
Total generated tokens:                  201469
Request throughput (req/s):              7.51
Output token throughput (tok/s):         1512.21
Peak output token throughput (tok/s):    2821.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          3143.94
---------------Time to First Token----------------
Mean TTFT (ms):                          43963.16
Median TTFT (ms):                        41177.19
P99 TTFT (ms):                           91086.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.09
Median TPOT (ms):                        122.07
P99 TPOT (ms):                           269.65
---------------Inter-token Latency----------------
Mean ITL (ms):                           112.90
Median ITL (ms):                         78.96
P99 ITL (ms):                            280.07
==================================================

MindIE

推論サービスのスクリプト 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve Qwen/Qwen3-8B
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  74.36
Total input tokens:                      217393
Total generated tokens:                  201609
Request throughput (req/s):              13.45
Output token throughput (tok/s):         2711.10
Peak output token throughput (tok/s):    4588.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5634.45
---------------Time to First Token----------------
Mean TTFT (ms):                          8242.22
Median TTFT (ms):                        8298.00
P99 TTFT (ms):                           18074.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          788.31
Median TPOT (ms):                        204.41
P99 TPOT (ms):                           6547.97
---------------Inter-token Latency----------------
Mean ITL (ms):                           155.96
Median ITL (ms):                         45.35
P99 ITL (ms):                            218.57
==================================================

結果： MineIE (5634.4 tok/s) > vLLM (3143.94 tok/s)

2. 量子化：MindIE

推論サービスのスクリプト 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  69.32
Total input tokens:                      217393
Total generated tokens:                  201315
Request throughput (req/s):              14.42
Output token throughput (tok/s):         2903.93
Peak output token throughput (tok/s):    4799.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6039.79
---------------Time to First Token----------------
Mean TTFT (ms):                          5834.61
Median TTFT (ms):                        5648.87
P99 TTFT (ms):                           11181.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          822.87
Median TPOT (ms):                        188.93
P99 TPOT (ms):                           7710.20
---------------Inter-token Latency----------------
Mean ITL (ms):                           148.97
Median ITL (ms):                         45.31
P99 ITL (ms):                            97.87
==================================================

3. プレフィックスキャッシュの有効化：MindIE

推論サービスのスクリプト 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  66.96
Total input tokens:                      217393
Total generated tokens:                  201517
Request throughput (req/s):              14.93
Output token throughput (tok/s):         3009.64
Peak output token throughput (tok/s):    4985.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6256.39
---------------Time to First Token----------------
Mean TTFT (ms):                          5320.32
Median TTFT (ms):                        5159.10
P99 TTFT (ms):                           10384.83
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          796.13
Median TPOT (ms):                        183.35
P99 TPOT (ms):                           7267.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           141.91
Median ITL (ms):                         43.55
P99 ITL (ms):                            115.20
==================================================

4. バッチサイズの調整：MindIE

推論サービスのスクリプト 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching --max-batch-size=400 --max-prefill-batch-size=200
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  69.07
Total input tokens:                      217393
Total generated tokens:                  201424
Request throughput (req/s):              14.48
Output token throughput (tok/s):         2916.34
Peak output token throughput (tok/s):    6017.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6063.89
---------------Time to First Token----------------
Mean TTFT (ms):                          6335.34
Median TTFT (ms):                        5812.45
P99 TTFT (ms):                           11260.65
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          581.93
Median TPOT (ms):                        167.14
P99 TPOT (ms):                           5501.66
---------------Inter-token Latency----------------
Mean ITL (ms):                           143.92
Median ITL (ms):                         70.27
P99 ITL (ms):                            281.16
==================================================

最適化項目の一覧

最適化項目 スループット改善率
エンジンの選択 +79.2%
量子化 +7.2%
プレフィックスキャッシュ +3.6%
バッチサイズの調整 -

その他のベンチマークケース

さらに、さまざまなワークロードに対する汎用性を評価するため、最適化済み設定でベンチマークを実施しました。

基準設定の推論スクリプト 
1
vllm serve Qwen/Qwen3-8B
基準設定のベンチマーク結果 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  1110.48
Total input tokens:                      3200000
Total generated tokens:                  10000
Request throughput (req/s):              0.09
Output token throughput (tok/s):         9.01
Peak output token throughput (tok/s):    147.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          2890.64
---------------Time to First Token----------------
Mean TTFT (ms):                          559550.01
Median TTFT (ms):                        561560.86
P99 TTFT (ms):                           1096851.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          344.18
Median TPOT (ms):                        346.89
P99 TPOT (ms):                           668.43
---------------Inter-token Latency----------------
Mean ITL (ms):                           340.74
Median ITL (ms):                         48.54
P99 ITL (ms):                            59.04
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  273.17
Total input tokens:                      1997463
Total generated tokens:                  99812
Request throughput (req/s):              1.83
Output token throughput (tok/s):         365.38
Peak output token throughput (tok/s):    1062.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          7677.41
---------------Time to First Token----------------
Mean TTFT (ms):                          132541.20
Median TTFT (ms):                        135284.67
P99 TTFT (ms):                           262452.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          99.26
Median TPOT (ms):                        100.01
P99 TPOT (ms):                           175.24
---------------Inter-token Latency----------------
Mean ITL (ms):                           98.69
Median ITL (ms):                         55.84
P99 ITL (ms):                            134.52
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  109.30
Total input tokens:                      998057
Total generated tokens:                  50000
Request throughput (req/s):              4.57
Output token throughput (tok/s):         457.47
Peak output token throughput (tok/s):    1904.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          9589.00
---------------Time to First Token----------------
Mean TTFT (ms):                          53978.52
Median TTFT (ms):                        57661.39
P99 TTFT (ms):                           103428.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          140.39
Median TPOT (ms):                        142.80
P99 TPOT (ms):                           275.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           139.05
Median ITL (ms):                         63.53
P99 ITL (ms):                            627.10
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  22.59
Total input tokens:                      127782
Total generated tokens:                  4000
Request throughput (req/s):              44.28
Output token throughput (tok/s):         177.10
Peak output token throughput (tok/s):    1280.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5834.79
---------------Time to First Token----------------
Mean TTFT (ms):                          18464.39
Median TTFT (ms):                        17489.55
P99 TTFT (ms):                           22134.55
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.55
Median TPOT (ms):                        135.76
P99 TPOT (ms):                           149.27
---------------Inter-token Latency----------------
Mean ITL (ms):                           94.16
Median ITL (ms):                         83.83
P99 ITL (ms):                            251.91
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  275.57
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              0.36
Output token throughput (tok/s):         80.06
Peak output token throughput (tok/s):    92.00
Peak concurrent requests:                8.00
Total Token throughput (tok/s):          164.46
---------------Time to First Token----------------
Mean TTFT (ms):                          145.88
Median TTFT (ms):                        142.78
P99 TTFT (ms):                           201.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          47.28
Median TPOT (ms):                        47.29
P99 TPOT (ms):                           50.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           47.10
Median ITL (ms):                         46.72
P99 ITL (ms):                            91.82
==================================================
最適化後の推論スクリプト 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching
最適化後のベンチマーク結果 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  296.36
Total input tokens:                      3200000
Total generated tokens:                  9693
Request throughput (req/s):              0.34
Output token throughput (tok/s):         32.71
Peak output token throughput (tok/s):    330.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          10830.52
---------------Time to First Token----------------
Mean TTFT (ms):                          147133.58
Median TTFT (ms):                        147851.57
P99 TTFT (ms):                           289974.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          242.08
Median TPOT (ms):                        232.74
P99 TPOT (ms):                           741.49
---------------Inter-token Latency----------------
Mean ITL (ms):                           225.71
Median ITL (ms):                         46.46
P99 ITL (ms):                            56.81
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  157.15
Total input tokens:                      2000000
Total generated tokens:                  99961
Request throughput (req/s):              3.18
Output token throughput (tok/s):         636.07
Peak output token throughput (tok/s):    2142.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          13362.49
---------------Time to First Token----------------
Mean TTFT (ms):                          70716.71
Median TTFT (ms):                        74982.88
P99 TTFT (ms):                           148427.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          130.97
Median TPOT (ms):                        134.98
P99 TPOT (ms):                           208.16
---------------Inter-token Latency----------------
Mean ITL (ms):                           130.95
Median ITL (ms):                         63.84
P99 ITL (ms):                            489.52
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  67.34
Total input tokens:                      1000000
Total generated tokens:                  49975
Request throughput (req/s):              7.42
Output token throughput (tok/s):         742.11
Peak output token throughput (tok/s):    3000.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          15591.71
---------------Time to First Token----------------
Mean TTFT (ms):                          29078.68
Median TTFT (ms):                        32203.66
P99 TTFT (ms):                           62077.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          194.54
Median TPOT (ms):                        192.07
P99 TPOT (ms):                           358.38
---------------Inter-token Latency----------------
Mean ITL (ms):                           194.46
Median ITL (ms):                         68.54
P99 ITL (ms):                            4496.71
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  8.78
Total input tokens:                      128000
Total generated tokens:                  4000
Request throughput (req/s):              113.94
Output token throughput (tok/s):         455.76
Peak output token throughput (tok/s):    2511.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          15040.15
---------------Time to First Token----------------
Mean TTFT (ms):                          4491.88
Median TTFT (ms):                        4467.47
P99 TTFT (ms):                           7693.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1249.72
Median TPOT (ms):                        1258.92
P99 TPOT (ms):                           2252.88
---------------Inter-token Latency----------------
Mean ITL (ms):                           1249.72
Median ITL (ms):                         54.67
P99 ITL (ms):                            6378.56
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             4
Benchmark duration (s):                  1292.33
Total input tokens:                      217393
Total generated tokens:                  201773
Request throughput (req/s):              0.77
Output token throughput (tok/s):         156.13
Peak output token throughput (tok/s):    200.00
Peak concurrent requests:                9.00
Total Token throughput (tok/s):          324.35
---------------Time to First Token----------------
Mean TTFT (ms):                          112.38
Median TTFT (ms):                        111.05
P99 TTFT (ms):                           154.06
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.42
Median TPOT (ms):                        24.47
P99 TPOT (ms):                           35.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.08
Median ITL (ms):                         23.05
P99 ITL (ms):                            79.54
==================================================