Ana içeriğe geç

ASCEND 910B NPU üzerinde Qwen3-8B işlem hacmi optimizasyonu

Sonuç

qwen3-8b-910b

ASCEND 910B NPU üzerinde Qwen3-8B işlem hacmini optimize etmek için önerilen yapılandırma:

Model sunum komutu 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching

Optimizasyon öncesi ve sonrası benchmark sonuçlarının karşılaştırması:

Benchmark senaryosu temel yapılandırma (optimizasyonsuz vLLM) Optimize edilmiş
ShareGPT Toplam TPS: 3143.94
Ortalama TPOT(ms): 27.61		 Toplam TPS: 6256.39 (+99.0%)
Ortalama TPOT(ms): 796.13
Kısa istem Toplam TPS: 5834.79
Ortalama TPOT(ms): 125.55		 Toplam TPS: 16611.86 (+184.7%)
Ortalama TPOT(ms): 1131.26
Orta uzunlukta istem Toplam TPS: 9589.00
Ortalama TPOT(ms): 140.39		 Toplam TPS: 14484.86 (+51.1%)
Ortalama TPOT(ms): 122.59
Uzun istem Toplam TPS: 7677.41
Ortalama TPOT(ms): 99.26		 Toplam TPS: 11159.01 (+45.4%)
Ortalama TPOT(ms): 97.74
Çok uzun istem Toplam TPS: 2890.64
Ortalama TPOT(ms): 344.18		 Toplam TPS: 7835.01 (+171.1%)
Ortalama TPOT(ms): 169.70

Not

  1. Benchmark testlerimiz olası tüm optimizasyon birleşimlerini kapsamaz. Örneğin, varsayılan yapılandırmasında en iyi performansı gösteren çıkarım motorunu sonraki ayarlamalar için başlangıç noktası olarak seçeriz. Bu eleme yaklaşımı yerel bir optimum verir; bu, genel optimum olmayabilir.
  2. En büyük toplu iş boyutu, zamanlama yapılandırması, genişletilmiş KV önbelleği, CUDA graph, Torch Compile gibi belirli kullanım senaryolarına bağlı başka optimizasyonlar da vardır. Bu belgedeki sonuçlar, daha hedefli optimizasyonlar için başlangıç noktası olabilir.
  3. Testler belirli donanım ve yazılım yapılandırmalarında yürütülmüştür. Çıkarım motorlarındaki gelişmeler sonuçları değiştirebilir.

Eksik noktalar veya yeni değişiklikleri yansıtacak güncellemeler varsa bize bildirin.

Optimizasyon hedefi

Yüksek eş zamanlı isteklerde yüksek işlem hacmi sağlar.

Deney kurulumu

Model

Qwen/Qwen3-30B-A3B

Donanım

Tek Ascend 910B NPU.

Motor sürümü

  • vLLM-Ascend: v0.9.1
  • MindIE: 2.1RC1

Benchmark veri kümesi

  1. ShareGPT
  2. Farklı dizi uzunluklarına sahip rastgele veri kümesi:
    • Çok uzun istem: 32000 giriş token'ı, 100 çıkış token'ı
    • Uzun istem: 4000 giriş token'ı, 200 çıkış token'ı
    • Orta uzunlukta istem: 2000 giriş token'ı, 100 çıkış token'ı
    • Kısa istem: 128 giriş token'ı, 4 çıkış token'ı

Benchmark betiği

Model performansını ölçmek için vLLM bench CLI aracını kullanıyoruz. Benchmark şu komutla çalıştırılır:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-8B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-8B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Deney sonuçları

1. Çıkarım motoru seçimi

vLLM

Model sunum betiği 
1
vllm serve Qwen/Qwen3-8B
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  133.23
Total input tokens:                      217393
Total generated tokens:                  201469
Request throughput (req/s):              7.51
Output token throughput (tok/s):         1512.21
Peak output token throughput (tok/s):    2821.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          3143.94
---------------Time to First Token----------------
Mean TTFT (ms):                          43963.16
Median TTFT (ms):                        41177.19
P99 TTFT (ms):                           91086.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.09
Median TPOT (ms):                        122.07
P99 TPOT (ms):                           269.65
---------------Inter-token Latency----------------
Mean ITL (ms):                           112.90
Median ITL (ms):                         78.96
P99 ITL (ms):                            280.07
==================================================

MindIE

Model sunum betiği 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve Qwen/Qwen3-8B
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  74.36
Total input tokens:                      217393
Total generated tokens:                  201609
Request throughput (req/s):              13.45
Output token throughput (tok/s):         2711.10
Peak output token throughput (tok/s):    4588.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5634.45
---------------Time to First Token----------------
Mean TTFT (ms):                          8242.22
Median TTFT (ms):                        8298.00
P99 TTFT (ms):                           18074.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          788.31
Median TPOT (ms):                        204.41
P99 TPOT (ms):                           6547.97
---------------Inter-token Latency----------------
Mean ITL (ms):                           155.96
Median ITL (ms):                         45.35
P99 ITL (ms):                            218.57
==================================================

Sonuç: MineIE (5634.4 tok/s) > vLLM (3143.94 tok/s)

2. MindIE için Quantization

Model sunum betiği 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  69.32
Total input tokens:                      217393
Total generated tokens:                  201315
Request throughput (req/s):              14.42
Output token throughput (tok/s):         2903.93
Peak output token throughput (tok/s):    4799.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6039.79
---------------Time to First Token----------------
Mean TTFT (ms):                          5834.61
Median TTFT (ms):                        5648.87
P99 TTFT (ms):                           11181.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          822.87
Median TPOT (ms):                        188.93
P99 TPOT (ms):                           7710.20
---------------Inter-token Latency----------------
Mean ITL (ms):                           148.97
Median ITL (ms):                         45.31
P99 ITL (ms):                            97.87
==================================================

3. MindIE için Önek önbelleğini etkinleştirme

Model sunum betiği 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  66.96
Total input tokens:                      217393
Total generated tokens:                  201517
Request throughput (req/s):              14.93
Output token throughput (tok/s):         3009.64
Peak output token throughput (tok/s):    4985.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6256.39
---------------Time to First Token----------------
Mean TTFT (ms):                          5320.32
Median TTFT (ms):                        5159.10
P99 TTFT (ms):                           10384.83
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          796.13
Median TPOT (ms):                        183.35
P99 TPOT (ms):                           7267.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           141.91
Median ITL (ms):                         43.55
P99 ITL (ms):                            115.20
==================================================

4. MindIE için Toplu iş boyutunu ayarlama

Model sunum betiği 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching --max-batch-size=400 --max-prefill-batch-size=200
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  69.07
Total input tokens:                      217393
Total generated tokens:                  201424
Request throughput (req/s):              14.48
Output token throughput (tok/s):         2916.34
Peak output token throughput (tok/s):    6017.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6063.89
---------------Time to First Token----------------
Mean TTFT (ms):                          6335.34
Median TTFT (ms):                        5812.45
P99 TTFT (ms):                           11260.65
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          581.93
Median TPOT (ms):                        167.14
P99 TPOT (ms):                           5501.66
---------------Inter-token Latency----------------
Mean ITL (ms):                           143.92
Median ITL (ms):                         70.27
P99 ITL (ms):                            281.16
==================================================

Optimizasyon seçeneklerinin özeti

Optimizasyon seçeneği İşlem hacmi iyileşmesi
Motor seçimi +79.2%
Quantization +7.2%
Önek önbellekleme +3.6%
Toplu iş boyutunu ayarlama -

Diğer benchmark senaryoları

Optimize edilmiş yapılandırmanın farklı iş yüklerine genellenebilirliğini değerlendirmek için ek benchmark testleri yaptık.

Temel model sunum betiği 
1
vllm serve Qwen/Qwen3-8B
Temel benchmark sonuçları 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  1110.48
Total input tokens:                      3200000
Total generated tokens:                  10000
Request throughput (req/s):              0.09
Output token throughput (tok/s):         9.01
Peak output token throughput (tok/s):    147.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          2890.64
---------------Time to First Token----------------
Mean TTFT (ms):                          559550.01
Median TTFT (ms):                        561560.86
P99 TTFT (ms):                           1096851.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          344.18
Median TPOT (ms):                        346.89
P99 TPOT (ms):                           668.43
---------------Inter-token Latency----------------
Mean ITL (ms):                           340.74
Median ITL (ms):                         48.54
P99 ITL (ms):                            59.04
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  273.17
Total input tokens:                      1997463
Total generated tokens:                  99812
Request throughput (req/s):              1.83
Output token throughput (tok/s):         365.38
Peak output token throughput (tok/s):    1062.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          7677.41
---------------Time to First Token----------------
Mean TTFT (ms):                          132541.20
Median TTFT (ms):                        135284.67
P99 TTFT (ms):                           262452.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          99.26
Median TPOT (ms):                        100.01
P99 TPOT (ms):                           175.24
---------------Inter-token Latency----------------
Mean ITL (ms):                           98.69
Median ITL (ms):                         55.84
P99 ITL (ms):                            134.52
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  109.30
Total input tokens:                      998057
Total generated tokens:                  50000
Request throughput (req/s):              4.57
Output token throughput (tok/s):         457.47
Peak output token throughput (tok/s):    1904.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          9589.00
---------------Time to First Token----------------
Mean TTFT (ms):                          53978.52
Median TTFT (ms):                        57661.39
P99 TTFT (ms):                           103428.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          140.39
Median TPOT (ms):                        142.80
P99 TPOT (ms):                           275.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           139.05
Median ITL (ms):                         63.53
P99 ITL (ms):                            627.10
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  22.59
Total input tokens:                      127782
Total generated tokens:                  4000
Request throughput (req/s):              44.28
Output token throughput (tok/s):         177.10
Peak output token throughput (tok/s):    1280.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5834.79
---------------Time to First Token----------------
Mean TTFT (ms):                          18464.39
Median TTFT (ms):                        17489.55
P99 TTFT (ms):                           22134.55
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.55
Median TPOT (ms):                        135.76
P99 TPOT (ms):                           149.27
---------------Inter-token Latency----------------
Mean ITL (ms):                           94.16
Median ITL (ms):                         83.83
P99 ITL (ms):                            251.91
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  275.57
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              0.36
Output token throughput (tok/s):         80.06
Peak output token throughput (tok/s):    92.00
Peak concurrent requests:                8.00
Total Token throughput (tok/s):          164.46
---------------Time to First Token----------------
Mean TTFT (ms):                          145.88
Median TTFT (ms):                        142.78
P99 TTFT (ms):                           201.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          47.28
Median TPOT (ms):                        47.29
P99 TPOT (ms):                           50.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           47.10
Median ITL (ms):                         46.72
P99 ITL (ms):                            91.82
==================================================
Optimize edilmiş model sunum betiği 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching
Optimize edilmiş benchmark sonuçları 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  296.36
Total input tokens:                      3200000
Total generated tokens:                  9693
Request throughput (req/s):              0.34
Output token throughput (tok/s):         32.71
Peak output token throughput (tok/s):    330.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          10830.52
---------------Time to First Token----------------
Mean TTFT (ms):                          147133.58
Median TTFT (ms):                        147851.57
P99 TTFT (ms):                           289974.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          242.08
Median TPOT (ms):                        232.74
P99 TPOT (ms):                           741.49
---------------Inter-token Latency----------------
Mean ITL (ms):                           225.71
Median ITL (ms):                         46.46
P99 ITL (ms):                            56.81
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  157.15
Total input tokens:                      2000000
Total generated tokens:                  99961
Request throughput (req/s):              3.18
Output token throughput (tok/s):         636.07
Peak output token throughput (tok/s):    2142.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          13362.49
---------------Time to First Token----------------
Mean TTFT (ms):                          70716.71
Median TTFT (ms):                        74982.88
P99 TTFT (ms):                           148427.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          130.97
Median TPOT (ms):                        134.98
P99 TPOT (ms):                           208.16
---------------Inter-token Latency----------------
Mean ITL (ms):                           130.95
Median ITL (ms):                         63.84
P99 ITL (ms):                            489.52
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  67.34
Total input tokens:                      1000000
Total generated tokens:                  49975
Request throughput (req/s):              7.42
Output token throughput (tok/s):         742.11
Peak output token throughput (tok/s):    3000.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          15591.71
---------------Time to First Token----------------
Mean TTFT (ms):                          29078.68
Median TTFT (ms):                        32203.66
P99 TTFT (ms):                           62077.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          194.54
Median TPOT (ms):                        192.07
P99 TPOT (ms):                           358.38
---------------Inter-token Latency----------------
Mean ITL (ms):                           194.46
Median ITL (ms):                         68.54
P99 ITL (ms):                            4496.71
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  8.78
Total input tokens:                      128000
Total generated tokens:                  4000
Request throughput (req/s):              113.94
Output token throughput (tok/s):         455.76
Peak output token throughput (tok/s):    2511.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          15040.15
---------------Time to First Token----------------
Mean TTFT (ms):                          4491.88
Median TTFT (ms):                        4467.47
P99 TTFT (ms):                           7693.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1249.72
Median TPOT (ms):                        1258.92
P99 TPOT (ms):                           2252.88
---------------Inter-token Latency----------------
Mean ITL (ms):                           1249.72
Median ITL (ms):                         54.67
P99 ITL (ms):                            6378.56
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             4
Benchmark duration (s):                  1292.33
Total input tokens:                      217393
Total generated tokens:                  201773
Request throughput (req/s):              0.77
Output token throughput (tok/s):         156.13
Peak output token throughput (tok/s):    200.00
Peak concurrent requests:                9.00
Total Token throughput (tok/s):          324.35
---------------Time to First Token----------------
Mean TTFT (ms):                          112.38
Median TTFT (ms):                        111.05
P99 TTFT (ms):                           154.06
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.42
Median TPOT (ms):                        24.47
P99 TPOT (ms):                           35.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.08
Median ITL (ms):                         23.05
P99 ITL (ms):                            79.54
==================================================