Ana içeriğe geç

NVIDIA H100 GPU üzerinde Qwen3-235B-A22B işlem hacmi optimizasyonu

Sonuç

qwen3-235b-a22b-h100

H100 GPU üzerinde Qwen3-235B-A22B-Instruct-2507 işlem hacmini optimize etmek için önerilen yapılandırma:

Model sunum komutu 
1
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 -tp 4 --max-num-batched-tokens 16384

Optimizasyon öncesi ve sonrası benchmark sonuçlarının karşılaştırması:

Benchmark senaryosu temel yapılandırma (optimizasyonsuz vLLM) Optimize edilmiş
ShareGPT Toplam TPS: 8424.86
Ortalama TPOT(ms): 147.56		 Toplam TPS: 7787.17x2 (+84.9%)
Ortalama TPOT(ms): 186.26
Kısa istem Toplam TPS: 14668.27
Ortalama TPOT(ms): 212.98		 Toplam TPS: 21211.83x2 (+189.2%)
Ortalama TPOT(ms): 125.18
Orta uzunlukta istem Toplam TPS: 23858.69
Ortalama TPOT(ms): 101.08		 Toplam TPS: 18950.04x2 (+58.9%)
Ortalama TPOT(ms): 125.93
Uzun istem Toplam TPS: 21113.53
Ortalama TPOT(ms): 68.94		 Toplam TPS: 16478.79x2 (+56.1%)
Ortalama TPOT(ms): 89.52
Çok uzun istem Toplam TPS: 22657.51
Ortalama TPOT(ms): 132.10		 Toplam TPS: 15366.32x2 (+35.6%)
Ortalama TPOT(ms): 194.60

Not

  1. Benchmark testlerimiz olası tüm optimizasyon birleşimlerini kapsamaz. Örneğin, varsayılan yapılandırmasında en iyi performansı gösteren çıkarım motorunu sonraki ayarlamalar için başlangıç noktası olarak seçeriz. Bu eleme yaklaşımı yerel bir optimum verir; bu, genel optimum olmayabilir.
  2. En büyük toplu iş boyutu, zamanlama yapılandırması, genişletilmiş KV önbelleği, CUDA graph, Torch Compile gibi belirli kullanım senaryolarına bağlı başka optimizasyonlar da vardır. Bu belgedeki sonuçlar, daha hedefli optimizasyonlar için başlangıç noktası olabilir.
  3. Testler belirli donanım ve yazılım yapılandırmalarında yürütülmüştür. Çıkarım motorlarındaki gelişmeler sonuçları değiştirebilir.
  4. Nicemleme doğruluğu etkileyebilir; ancak FP8 nicemleme çoğu modelde %1'den az doğruluk kaybı sağlayabilir. Ayrıntılar için değerlendirme sonuçlarına bakın. Bu nedenle yüksek işlem hacimli sunum senaryolarında FP8 nicemleme önemle önerilir.

Eksik noktalar veya yeni değişiklikleri yansıtacak güncellemeler varsa bize bildirin.

Optimizasyon hedefi

Yüksek eş zamanlı isteklerde yüksek işlem hacmi sağlar.

Deney kurulumu

Model

Qwen3-235B-A22B-Instruct-2507

Donanım

Tek düğümde 8 × NVIDIA H100 SXM GPU.

Motor sürümü

  • vLLM: v0.11.0
  • SGLang: v0.5.5.post1

Benchmark veri kümesi

  1. ShareGPT
  2. Farklı dizi uzunluklarına sahip rastgele veri kümesi:
    • Çok uzun istem: 32000 giriş token'ı, 100 çıkış token'ı
    • Uzun istem: 4000 giriş token'ı, 200 çıkış token'ı
    • Orta uzunlukta istem: 2000 giriş token'ı, 100 çıkış token'ı
    • Kısa istem: 128 giriş token'ı, 4 çıkış token'ı

Benchmark betiği

Model performansını ölçmek için vLLM bench CLI aracını kullanıyoruz. Benchmark şu komutla çalıştırılır:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-235B-A22B-Instruct-2507 --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-235B-A22B-Instruct-2507 --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Deney sonuçları

1. Çıkarım motoru seçimi

vLLM

Model sunum betiği 
1
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 -tp 8
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     984
Benchmark duration (s):                  48.28
Total input tokens:                      212894
Total generated tokens:                  193865
Request throughput (req/s):              20.38
Output token throughput (tok/s):         4015.36
Peak output token throughput (tok/s):    9638.00
Peak concurrent requests:                984.00
Total Token throughput (tok/s):          8424.86
---------------Time to First Token----------------
Mean TTFT (ms):                          8730.69
Median TTFT (ms):                        8707.89
P99 TTFT (ms):                           12940.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          147.56
Median TPOT (ms):                        114.67
P99 TPOT (ms):                           342.35
---------------Inter-token Latency----------------
Mean ITL (ms):                           81.31
Median ITL (ms):                         51.14
P99 ITL (ms):                            677.61
==================================================

SGLang

Model sunum betiği 
1
python3 -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-Instruct-2507 --tp-size 8
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     984
Benchmark duration (s):                  53.18
Total input tokens:                      214137
Total generated tokens:                  193829
Request throughput (req/s):              18.50
Output token throughput (tok/s):         3644.72
Peak output token throughput (tok/s):    8791.00
Peak concurrent requests:                984.00
Total Token throughput (tok/s):          7671.31
---------------Time to First Token----------------
Mean TTFT (ms):                          11715.28
Median TTFT (ms):                        6274.12
P99 TTFT (ms):                           32860.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          178.35
Median TPOT (ms):                        102.70
P99 TPOT (ms):                           841.80
---------------Inter-token Latency----------------
Mean ITL (ms):                           88.35
Median ITL (ms):                         44.39
P99 ITL (ms):                            732.90
==================================================

Sonuç: vLLM (8424.86 tok/s) > SGLang (7671.31 tok/s)

2. vLLM için Quantization

Model sunum betiği 
1
2
# TP8 only is not compatible with the model weight. Use EP as well.
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 -tp 8 --enable-expert-parallel
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     984
Benchmark duration (s):                  49.88
Total input tokens:                      211761
Total generated tokens:                  192350
Request throughput (req/s):              19.73
Output token throughput (tok/s):         3856.59
Peak output token throughput (tok/s):    9788.00
Peak concurrent requests:                984.00
Total Token throughput (tok/s):          8102.36
---------------Time to First Token----------------
Mean TTFT (ms):                          9755.64
Median TTFT (ms):                        9742.53
P99 TTFT (ms):                           14508.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          169.52
Median TPOT (ms):                        129.08
P99 TPOT (ms):                           353.11
---------------Inter-token Latency----------------
Mean ITL (ms):                           88.39
Median ITL (ms):                         49.03
P99 ITL (ms):                            343.14
==================================================

3. vLLM için Paralellik

TP4

Model sunum betiği 
1
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 -tp 4
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     984
Benchmark duration (s):                  54.12
Total input tokens:                      212992
Total generated tokens:                  194446
Request throughput (req/s):              18.18
Output token throughput (tok/s):         3592.57
Peak output token throughput (tok/s):    8605.00
Peak concurrent requests:                984.00
Total Token throughput (tok/s):          7527.80
---------------Time to First Token----------------
Mean TTFT (ms):                          9924.69
Median TTFT (ms):                        9896.14
P99 TTFT (ms):                           15574.61
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          169.20
Median TPOT (ms):                        121.96
P99 TPOT (ms):                           372.71
---------------Inter-token Latency----------------
Mean ITL (ms):                           89.89
Median ITL (ms):                         57.15
P99 ITL (ms):                            351.02
==================================================

TP4+EP

Model sunum betiği 
1
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 -tp 4 --enable-expert-parallel
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     984
Benchmark duration (s):                  57.08
Total input tokens:                      214311
Total generated tokens:                  192752
Request throughput (req/s):              17.24
Output token throughput (tok/s):         3376.99
Peak output token throughput (tok/s):    8005.00
Peak concurrent requests:                984.00
Total Token throughput (tok/s):          7131.69
---------------Time to First Token----------------
Mean TTFT (ms):                          10706.10
Median TTFT (ms):                        10676.53
P99 TTFT (ms):                           17226.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          190.24
Median TPOT (ms):                        131.62
P99 TPOT (ms):                           432.31
---------------Inter-token Latency----------------
Mean ITL (ms):                           96.98
Median ITL (ms):                         60.65
P99 ITL (ms):                            408.09
==================================================

4. vLLM için Dikkat arka ucu

Varsayılan FlashAttention'dır.

FlashInfer

Model sunum betiği 
1
VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 -tp 4
Benchmark sonucu 
1
# Crash

XFormers

Model sunum betiği 
1
VLLM_ATTENTION_BACKEND=XFORMERS vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 -tp 4
Benchmark sonucu 
1
# Crash on inference

5. vLLM için Toplu işteki en fazla token sayısı

Model sunum betiği 
1
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 -tp 4 --max-num-batched-tokens 16384
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# --max-num-batched-tokens 16384
============ Serving Benchmark Result ============
Successful requests:                     984
Benchmark duration (s):                  52.46
Total input tokens:                      214381
Total generated tokens:                  194148
Request throughput (req/s):              18.76
Output token throughput (tok/s):         3700.75
Peak output token throughput (tok/s):    8574.00
Peak concurrent requests:                984.00
Total Token throughput (tok/s):          7787.17
---------------Time to First Token----------------
Mean TTFT (ms):                          9600.73
Median TTFT (ms):                        9525.78
P99 TTFT (ms):                           13887.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          186.26
Median TPOT (ms):                        108.31
P99 TPOT (ms):                           634.48
---------------Inter-token Latency----------------
Mean ITL (ms):                           86.62
Median ITL (ms):                         56.97
P99 ITL (ms):                            632.37
==================================================

# --max-num-batched-tokens 32768
# Crash on inference

Optimizasyon seçeneklerinin özeti

Optimizasyon seçeneği İşlem hacmi iyileşmesi
Motor seçimi -
Quantization -
Paralellik +78.7%
Dikkat arka ucu -
Toplu işteki en fazla token sayısı +3.4%

Diğer benchmark senaryoları

Optimize edilmiş yapılandırmanın farklı iş yüklerine genellenebilirliğini değerlendirmek için ek benchmark testleri yaptık.

Temel model sunum betiği 
1
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 -tp 8
Temel benchmark sonuçları 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  141.60
Total input tokens:                      3200000
Total generated tokens:                  8191
Request throughput (req/s):              0.71
Output token throughput (tok/s):         57.85
Peak output token throughput (tok/s):    346.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          22657.51
---------------Time to First Token----------------
Mean TTFT (ms):                          70409.55
Median TTFT (ms):                        70737.29
P99 TTFT (ms):                           138084.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          132.10
Median TPOT (ms):                        134.49
P99 TPOT (ms):                           182.76
---------------Inter-token Latency----------------
Mean ITL (ms):                           129.44
Median ITL (ms):                         23.29
P99 ITL (ms):                            376.34
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  98.99
Total input tokens:                      1997942
Total generated tokens:                  92185
Request throughput (req/s):              5.05
Output token throughput (tok/s):         931.21
Peak output token throughput (tok/s):    2450.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          21113.53
---------------Time to First Token----------------
Mean TTFT (ms):                          47811.58
Median TTFT (ms):                        48562.54
P99 TTFT (ms):                           93161.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          68.94
Median TPOT (ms):                        72.58
P99 TPOT (ms):                           92.94
---------------Inter-token Latency----------------
Mean ITL (ms):                           68.75
Median ITL (ms):                         30.18
P99 ITL (ms):                            245.68
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  43.91
Total input tokens:                      998175
Total generated tokens:                  49500
Request throughput (req/s):              11.39
Output token throughput (tok/s):         1127.26
Peak output token throughput (tok/s):    4021.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          23858.69
---------------Time to First Token----------------
Mean TTFT (ms):                          21313.79
Median TTFT (ms):                        20242.32
P99 TTFT (ms):                           40939.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          101.08
Median TPOT (ms):                        108.64
P99 TPOT (ms):                           125.24
---------------Inter-token Latency----------------
Mean ITL (ms):                           100.16
Median ITL (ms):                         35.94
P99 ITL (ms):                            244.64
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     984
Benchmark duration (s):                  8.83
Total input tokens:                      125650
Total generated tokens:                  3936
Request throughput (req/s):              111.38
Output token throughput (tok/s):         445.53
Peak output token throughput (tok/s):    1489.00
Peak concurrent requests:                984.00
Total Token throughput (tok/s):          14668.27
---------------Time to First Token----------------
Mean TTFT (ms):                          7042.17
Median TTFT (ms):                        6882.45
P99 TTFT (ms):                           8721.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          212.98
Median TPOT (ms):                        238.84
P99 TPOT (ms):                           242.28
---------------Inter-token Latency----------------
Mean ITL (ms):                           159.73
Median ITL (ms):                         235.21
P99 ITL (ms):                            250.35
==================================================
Optimize edilmiş model sunum betiği 
1
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 -tp 4 --max-num-batched-tokens 16384
Optimize edilmiş benchmark sonuçları 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  208.79
Total input tokens:                      3200000
Total generated tokens:                  8281
Request throughput (req/s):              0.48
Output token throughput (tok/s):         39.66
Peak output token throughput (tok/s):    277.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          15366.32
---------------Time to First Token----------------
Mean TTFT (ms):                          103754.50
Median TTFT (ms):                        103784.48
P99 TTFT (ms):                           205204.25
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          194.60
Median TPOT (ms):                        194.50
P99 TPOT (ms):                           277.94
---------------Inter-token Latency----------------
Mean ITL (ms):                           193.15
Median ITL (ms):                         27.60
P99 ITL (ms):                            1117.34
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  126.76
Total input tokens:                      1997942
Total generated tokens:                  90966
Request throughput (req/s):              3.94
Output token throughput (tok/s):         717.60
Peak output token throughput (tok/s):    2139.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          16478.79
---------------Time to First Token----------------
Mean TTFT (ms):                          60513.14
Median TTFT (ms):                        61048.35
P99 TTFT (ms):                           120001.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          89.52
Median TPOT (ms):                        94.78
P99 TPOT (ms):                           121.95
---------------Inter-token Latency----------------
Mean ITL (ms):                           89.17
Median ITL (ms):                         34.82
P99 ITL (ms):                            642.94
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  55.28
Total input tokens:                      998175
Total generated tokens:                  49471
Request throughput (req/s):              9.04
Output token throughput (tok/s):         894.84
Peak output token throughput (tok/s):    3723.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          18950.04
---------------Time to First Token----------------
Mean TTFT (ms):                          26860.95
Median TTFT (ms):                        26155.54
P99 TTFT (ms):                           51773.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.93
Median TPOT (ms):                        140.65
P99 TPOT (ms):                           152.49
---------------Inter-token Latency----------------
Mean ITL (ms):                           124.89
Median ITL (ms):                         39.21
P99 ITL (ms):                            632.84
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     984
Benchmark duration (s):                  6.11
Total input tokens:                      125650
Total generated tokens:                  3936
Request throughput (req/s):              161.07
Output token throughput (tok/s):         644.28
Peak output token throughput (tok/s):    3676.00
Peak concurrent requests:                984.00
Total Token throughput (tok/s):          21211.83
---------------Time to First Token----------------
Mean TTFT (ms):                          5621.51
Median TTFT (ms):                        5525.90
P99 TTFT (ms):                           5803.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.18
Median TPOT (ms):                        152.34
P99 TPOT (ms):                           182.84
---------------Inter-token Latency----------------
Mean ITL (ms):                           93.89
Median ITL (ms):                         110.55
P99 ITL (ms):                            243.41
==================================================