Qwen3.5-9B işlem hacmi optimizasyonu

Sonuç

NVIDIA H100 80GB HBM3 üzerinde Qwen/Qwen3.5-9B işlem hacmini optimize etmek için önerilen yapılandırma:

Model sunum komutu

vllm serve Qwen/Qwen3.5-9B \
    --reasoning-parser=qwen3 \
    --performance-mode=throughput \
    --max-model-len=32768 \
    --max-num-seqs=512

Optimizasyon öncesi ve sonrası benchmark sonuçlarının karşılaştırması:

Benchmark senaryosu	temel yapılandırma (optimizasyonsuz vLLM)	Optimize edilmiş
ShareGPT profili	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12	Toplam TPS: 11768.22 (+2.09%) Ortalama TPOT(ms): 15.85
İşlem hacmi profili	Toplam TPS: 29822.16 Ortalama TPOT(ms): 34.63	Toplam TPS: 34470.14 (+15.59%) Ortalama TPOT(ms): 36.97
Uzun bağlam profili	Toplam TPS: 32416.84 Ortalama TPOT(ms): 10.21	Toplam TPS: 32658.73 (+0.75%) Ortalama TPOT(ms): 9.97
Yoğun üretim profili	Toplam TPS: 2973.42 Ortalama TPOT(ms): 0.33	Toplam TPS: 3025.11 (+1.74%) Ortalama TPOT(ms): 0.12

Not

Benchmark testlerimiz olası tüm optimizasyon birleşimlerini kapsamaz. Örneğin, varsayılan yapılandırmasında en iyi performansı gösteren çıkarım motorunu sonraki ayarlamalar için başlangıç noktası olarak seçeriz. Bu eleme yaklaşımı yerel bir optimum verir; bu, genel optimum olmayabilir.
En büyük toplu iş boyutu, zamanlama yapılandırması, genişletilmiş KV önbelleği, CUDA graph gibi belirli kullanım senaryolarına bağlı başka optimizasyonlar da vardır. Bu belgedeki sonuçlar, daha hedefli optimizasyonlar için başlangıç noktası olabilir.
Testler belirli donanım ve yazılım yapılandırmalarında yürütülmüştür. Çıkarım motorlarındaki gelişmeler sonuçları değiştirebilir.
Nicemleme doğruluğu etkileyebilir; ancak FP8 nicemleme çoğu modelde %1'den az doğruluk kaybı sağlayabilir. Ayrıntılar için değerlendirme sonuçlarına bakın. Bu nedenle düşük gecikmeli sunum senaryolarında FP8 nicemleme önemle önerilir.
Spekülatif kod çözme, düşük eş zamanlılıktaki isteklerin gecikmesini önemli ölçüde azaltabilir. Ancak hızlanma, benchmark veri dağılımına ve taslak model seçimine bağlıdır. Örneğin burada seçilen taslak model İngilizce verilerle eğitilmiştir; diğer dillerde en iyi performansı vermeyebilir.

Eksik noktalar veya yeni değişiklikleri yansıtacak güncellemeler varsa bize bildirin.

Deney kurulumu

Model

Qwen/Qwen3.5-9B

Donanım

NVIDIA H100 80GB HBM3

Motor sürümü

vLLM v0.17.1
SGLang v0.5.9

Benchmark yöntemi

Bu projede model sunum iş yükleri için GPUStack'in tek tıklamalı benchmark özelliği kullanılır. Bu belgedeki benchmark testleri bu akışla yürütülmüştür.

GPUStack'in benchmark uygulaması guidellm üzerine, sarmalayıcı benchmark-runner projesi aracılığıyla kurulmuştur.

Aşağıda listelenen benchmark yapılandırmalarında model dağıtımı, benchmark işinin gönderilmesi ve sonuçların toplanması GPUStack tarafından yapılır.

Benchmark profilleri

ShareGPT

dataset_name: ShareGPT
request_rate: 1000
total_requests: 1000

İşlem hacmi

dataset_name: Random
dataset_input_tokens: 1024
dataset_output_tokens: 128
dataset_seed: 42
request_rate: 1000
total_requests: 1000

Uzun bağlam

dataset_name: Random
dataset_input_tokens: 32000
dataset_output_tokens: 100
dataset_seed: 42
request_rate: 1
total_requests: 100

Yoğun üretim

dataset_name: Random
dataset_input_tokens: 1000
dataset_output_tokens: 2000
dataset_seed: 42
request_rate: 1
total_requests: 200

Deney sonuçları

Çıkarım motoru seçimi

vLLM

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--max-model-len=32768

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.74
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.27
Output token throughput (tok/s):         5202.94
Peak output token throughput (tok/s):    20132659.20
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11527.14
----------------------Latency---------------------
Mean Latency(s):                          24.11
Median Latency(s):                        21.90
P99 Latency(s):                           51.16
---------------Time to First Token----------------
Mean TTFT (ms):                          1604.23
Median TTFT (ms):                        646.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5737.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.12
Median TPOT (ms):                        3.42
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.68
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.45
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            118.73
==================================================

SGLang

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--context-length=32768

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  96.01
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              10.42
Output token throughput (tok/s):         2933.60
Peak output token throughput (tok/s):    92831.68
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          6499.41
----------------------Latency---------------------
Mean Latency(s):                          39.08
Median Latency(s):                        41.18
P99 Latency(s):                           74.40
---------------Time to First Token----------------
Mean TTFT (ms):                          26217.15
Median TTFT (ms):                        34401.34
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           41101.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          138.86
Median TPOT (ms):                        112.19
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           591.70
---------------Inter-token Latency----------------
Mean ITL (ms):                           45.86
Median ITL (ms):                         49.31
P95 ITL (ms):                            N/A
P99 ITL (ms):                            59.71
==================================================

Özet: vLLM Toplam TPS = 11527.14, SGLang Toplam TPS = 6499.41. vLLM, 5027.72 tok/s daha hızlıdır (77.36%); Ortalama TPOT = 15.12 ms ile 138.86 ms, azalma: 123.74 ms (89.11%).

Önek önbelleği

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--enable-prefix-caching
--max-model-len=32768

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  56.47
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              17.71
Output token throughput (tok/s):         5066.42
Peak output token throughput (tok/s):    31242972.50
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11224.70
----------------------Latency---------------------
Mean Latency(s):                          24.90
Median Latency(s):                        22.40
P99 Latency(s):                           52.82
---------------Time to First Token----------------
Mean TTFT (ms):                          1995.45
Median TTFT (ms):                        1341.75
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           6549.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.66
Median TPOT (ms):                        5.21
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           130.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.60
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            120.90
==================================================

Spekülatif kod çözme

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--speculative-config={"method":"mtp","num_speculative_tokens":4}
--max-model-len=32768

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     514
Maximum request concurrency:             512
Benchmark duration (s):                  12.27
Total input tokens:                      23313
Total generated tokens:                  131530
Request throughput (req/s):              41.90
Output token throughput (tok/s):         12792.49
Peak output token throughput (tok/s):    13918875.50
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          15059.89
----------------------Latency---------------------
Mean Latency(s):                          11.77
Median Latency(s):                        11.81
P99 Latency(s):                           12.07
---------------Time to First Token----------------
Mean TTFT (ms):                          268.95
Median TTFT (ms):                        0.00
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           3010.96
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          42.50
Median TPOT (ms):                        31.35
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           214.99
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Performans modu

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.66
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.29
Output token throughput (tok/s):         5189.06
Peak output token throughput (tok/s):    30482939.47
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11496.39
----------------------Latency---------------------
Mean Latency(s):                          24.00
Median Latency(s):                        21.51
P95 Latency(s):                           N/A
P99 Latency(s):                           51.01
---------------Time to First Token----------------
Mean TTFT (ms):                          1824.24
Median TTFT (ms):                        982.10
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5550.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.41
Median TPOT (ms):                        4.53
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.10
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.96
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            115.72
==================================================

En fazla dizi sayısı

Dizi sayısı 512

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-seqs=512

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.89
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.22
Output token throughput (tok/s):         5174.68
Peak output token throughput (tok/s):    21197768.76
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11464.53
----------------------Latency---------------------
Mean Latency(s):                          24.11
Median Latency(s):                        21.96
P95 Latency(s):                           48.41
P99 Latency(s):                           51.17
---------------Time to First Token----------------
Mean TTFT (ms):                          1893.47
Median TTFT (ms):                        1233.85
P95 TTFT (ms):                           5659.91
P99 TTFT (ms):                           6222.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.18
Median TPOT (ms):                        4.48
P95 TPOT (ms):                           91.71
P99 TPOT (ms):                           127.60
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.48
Median ITL (ms):                         0.00
P95 ITL (ms):                            83.92
P99 ITL (ms):                            116.65
==================================================

Dizi sayısı 1024

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-seqs=1024

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.87
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.23
Output token throughput (tok/s):         5205.42
Peak output token throughput (tok/s):    22885338.88
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11532.64
----------------------Latency---------------------
Mean Latency(s):                          24.15
Median Latency(s):                        21.71
P95 Latency(s):                           48.49
P99 Latency(s):                           51.24
---------------Time to First Token----------------
Mean TTFT (ms):                          1864.39
Median TTFT (ms):                        1137.51
P95 TTFT (ms):                           5803.37
P99 TTFT (ms):                           6125.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.87
Median TPOT (ms):                        4.70
P95 TPOT (ms):                           93.65
P99 TPOT (ms):                           127.45
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.28
Median ITL (ms):                         0.00
P95 ITL (ms):                            87.39
P99 ITL (ms):                            116.13
==================================================

Dizi sayısı 2048

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--max-model-len=32768
--max-num-seqs=2048

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  55.28
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.09
Output token throughput (tok/s):         5140.19
Peak output token throughput (tok/s):    23553347.13
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11388.13
----------------------Latency---------------------
Mean Latency(s):                          24.34
Median Latency(s):                        22.16
P95 Latency(s):                           48.86
P99 Latency(s):                           51.63
---------------Time to First Token----------------
Mean TTFT (ms):                          1853.85
Median TTFT (ms):                        1166.25
P95 TTFT (ms):                           5543.84
P99 TTFT (ms):                           6126.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.91
Median TPOT (ms):                        4.35
P95 TPOT (ms):                           100.86
P99 TPOT (ms):                           129.26
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.36
Median ITL (ms):                         0.00
P95 ITL (ms):                            94.30
P99 ITL (ms):                            118.29
==================================================

Özet: Seqs 1024 Toplam TPS = 11532.64, Seqs 2048 Toplam TPS = 11388.13. Seqs 1024, 144.51 tok/s daha hızlıdır (1.27%); Ortalama TPOT = 16.87 ms ile 16.91 ms, azalma: 0.04 ms (0.25%).

En fazla toplu iş token'ı ve Performans modu

24k

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=24576

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.56
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.33
Output token throughput (tok/s):         5196.51
Peak output token throughput (tok/s):    32356059.43
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11512.90
----------------------Latency---------------------
Mean Latency(s):                          23.98
Median Latency(s):                        21.78
P99 Latency(s):                           50.95
---------------Time to First Token----------------
Mean TTFT (ms):                          1870.77
Median TTFT (ms):                        1771.31
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5297.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.83
Median TPOT (ms):                        4.86
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           125.06
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.22
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            113.92
==================================================

32k

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=32768

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.20
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.45
Output token throughput (tok/s):         5268.65
Peak output token throughput (tok/s):    29985425.61
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11672.72
----------------------Latency---------------------
Mean Latency(s):                          23.72
Median Latency(s):                        21.21
P99 Latency(s):                           50.44
---------------Time to First Token----------------
Mean TTFT (ms):                          1970.35
Median TTFT (ms):                        1344.71
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5206.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.53
Median TPOT (ms):                        5.04
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           125.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.56
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            111.59
==================================================

Özet: 32k Toplam TPS = 11672.72, 24k Toplam TPS = 11512.90. 32k, 159.82 tok/s daha hızlıdır (1.39%); Ortalama TPOT = 16.53 ms ile 15.83 ms, artış: 0.70 ms (4.43% daha yavaş).

En fazla dizi sayısı ve Performans modu

En fazla dizi sayısı 256

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=256

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  57.32
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              17.45
Output token throughput (tok/s):         4973.66
Peak output token throughput (tok/s):    30062067.31
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11019.18
----------------------Latency---------------------
Mean Latency(s):                          23.57
Median Latency(s):                        22.86
P99 Latency(s):                           49.24
---------------Time to First Token----------------
Mean TTFT (ms):                          11061.67
Median TTFT (ms):                        14983.14
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           18807.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          44.58
Median TPOT (ms):                        31.66
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           229.02
---------------Inter-token Latency----------------
Mean ITL (ms):                           5.29
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            54.12
==================================================

En fazla dizi sayısı 512

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  53.72
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.62
Output token throughput (tok/s):         5311.75
Peak output token throughput (tok/s):    23159852.52
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11768.22
----------------------Latency---------------------
Mean Latency(s):                          23.50
Median Latency(s):                        21.13
P99 Latency(s):                           50.04
---------------Time to First Token----------------
Mean TTFT (ms):                          1826.53
Median TTFT (ms):                        1004.67
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5582.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.85
Median TPOT (ms):                        4.49
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           123.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.39
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            112.55
==================================================

Özet: Max Num Seqs 512 Toplam TPS = 11768.22, Max Num Seqs 256 Toplam TPS = 11019.18. Max Num Seqs 512, 749.05 tok/s daha hızlıdır (6.80%); Ortalama TPOT = 15.85 ms ile 44.58 ms, azalma: 28.73 ms (64.45%).

En fazla dizi sayısı ve En fazla toplu iş token'ı

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=32768
--max-num-seqs=512

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  55.33
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.07
Output token throughput (tok/s):         5115.71
Peak output token throughput (tok/s):    34751436.57
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11333.89
----------------------Latency---------------------
Mean Latency(s):                          24.34
Median Latency(s):                        22.08
P99 Latency(s):                           51.69
---------------Time to First Token----------------
Mean TTFT (ms):                          1831.81
Median TTFT (ms):                        1345.90
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5161.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.39
Median TPOT (ms):                        4.48
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.14
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.92
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            116.48
==================================================

En fazla dizi sayısı ve En fazla toplu iş token'ı ve Performans modu

Toplu iş token sayısı 32k, dizi sayısı 512

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=32768
--max-num-seqs=512

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  55.33
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.07
Output token throughput (tok/s):         5115.71
Peak output token throughput (tok/s):    34751436.57
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11333.89
----------------------Latency---------------------
Mean Latency(s):                          24.34
Median Latency(s):                        22.08
P95 Latency(s):                           N/A
P99 Latency(s):                           51.69
---------------Time to First Token----------------
Mean TTFT (ms):                          1831.81
Median TTFT (ms):                        1345.90
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5161.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.39
Median TPOT (ms):                        4.48
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.14
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.92
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            116.48
==================================================

Toplu iş token sayısı 48k, dizi sayısı 512

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=49152
--max-num-seqs=512

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.48
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.36
Output token throughput (tok/s):         5223.82
Peak output token throughput (tok/s):    33459752.45
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11573.41
----------------------Latency---------------------
Mean Latency(s):                          23.96
Median Latency(s):                        21.63
P95 Latency(s):                           N/A
P99 Latency(s):                           50.92
---------------Time to First Token----------------
Mean TTFT (ms):                          2022.91
Median TTFT (ms):                        2480.42
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5146.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.20
Median TPOT (ms):                        5.63
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           124.01
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.04
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            111.00
==================================================

Toplu iş token sayısı 48k, dizi sayısı 768

Profil: ShareGPT

Arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-batched-tokens=49152
--max-num-seqs=768

Benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.07
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.49
Output token throughput (tok/s):         5248.67
Peak output token throughput (tok/s):    36318641.50
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11628.47
----------------------Latency---------------------
Mean Latency(s):                          23.72
Median Latency(s):                        21.75
P95 Latency(s):                           N/A
P99 Latency(s):                           50.48
---------------Time to First Token----------------
Mean TTFT (ms):                          1817.87
Median TTFT (ms):                        1821.59
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5041.22
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.65
Median TPOT (ms):                        4.57
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           123.59
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.23
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            114.09
==================================================

Özet: Batch Token 48k and Seqs 768 Toplam TPS = 11628.47, Batch Token 32k and Seqs 512 Toplam TPS = 11333.89. Batch Token 48k and Seqs 768, 294.57 tok/s daha hızlıdır (2.60%); Ortalama TPOT = 15.65 ms ile 16.39 ms, azalma: 0.74 ms (4.51%).

Optimizasyon seçeneklerinin özeti

Benchmark senaryoları	Optimize edilmiş	Temel yapılandırma
Çıkarım motoru seçimi	Toplam TPS: 11527.14 (+0.00%) Ortalama TPOT(ms): 15.12	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12
Önek önbelleği	Toplam TPS: 11224.70 (-2.62%) Ortalama TPOT(ms): 16.66	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12
Spekülatif kod çözme	Toplam TPS: 15059.89 (Başarı oranı: 51.5%, optimizasyon atlandı) Ortalama TPOT(ms): 42.50	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12
Performans modu	Toplam TPS: 11496.39 (-0.27%) Ortalama TPOT(ms): 16.41	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12
En fazla dizi sayısı	Toplam TPS: 11532.64 (+0.05%) Ortalama TPOT(ms): 16.87	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12
En fazla toplu iş token'ı && Performans modu	Toplam TPS: 11672.72 (+1.26%) Ortalama TPOT(ms): 16.53	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12
En fazla dizi sayısı && Performans modu	Toplam TPS: 11768.22 (+2.09%) Ortalama TPOT(ms): 15.85	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12
En fazla dizi sayısı && En fazla toplu iş token'ı	Toplam TPS: 11333.89 (-1.68%) Ortalama TPOT(ms): 16.39	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12
En fazla dizi sayısı && En fazla toplu iş token'ı && Performans modu	Toplam TPS: 11628.47 (+0.88%) Ortalama TPOT(ms): 15.65	Toplam TPS: 11527.14 Ortalama TPOT(ms): 15.12

Diğer benchmark senaryoları

ShareGPT profili

Temel arka uç parametreleri:

--reasoning-parser=qwen3
--max-model-len=32768

Temel benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  54.74
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.27
Output token throughput (tok/s):         5202.94
Peak output token throughput (tok/s):    20132659.20
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11527.14
----------------------Latency---------------------
Mean Latency(s):                          24.11
Median Latency(s):                        21.90
P99 Latency(s):                           51.16
---------------Time to First Token----------------
Mean TTFT (ms):                          1604.23
Median TTFT (ms):                        646.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5737.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.12
Median TPOT (ms):                        3.42
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           127.68
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.45
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            118.73
==================================================

Optimize edilmiş arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Optimize edilmiş benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  53.72
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              18.62
Output token throughput (tok/s):         5311.75
Peak output token throughput (tok/s):    23159852.52
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          11768.22
----------------------Latency---------------------
Mean Latency(s):                          23.50
Median Latency(s):                        21.13
P99 Latency(s):                           50.04
---------------Time to First Token----------------
Mean TTFT (ms):                          1826.53
Median TTFT (ms):                        1004.67
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           5582.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.85
Median TPOT (ms):                        4.49
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           123.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.39
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            112.55
==================================================

İşlem hacmi profili

Temel arka uç parametreleri:

--reasoning-parser=qwen3
--max-model-len=32768

Temel benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  43.00
Total input tokens:                      1033932
Total generated tokens:                  128000
Request throughput (req/s):              23.26
Output token throughput (tok/s):         3285.25
Peak output token throughput (tok/s):    4418690.63
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          29822.16
----------------------Latency---------------------
Mean Latency(s):                          21.27
Median Latency(s):                        21.11
P99 Latency(s):                           36.40
---------------Time to First Token----------------
Mean TTFT (ms):                          4433.02
Median TTFT (ms):                        1510.44
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           14914.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          34.63
Median TPOT (ms):                        11.80
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           116.52
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Optimize edilmiş arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Optimize edilmiş benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  39.10
Total input tokens:                      1033932
Total generated tokens:                  128000
Request throughput (req/s):              25.55
Output token throughput (tok/s):         3797.28
Peak output token throughput (tok/s):    7724761.32
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          34470.14
----------------------Latency---------------------
Mean Latency(s):                          19.38
Median Latency(s):                        19.24
P99 Latency(s):                           31.36
---------------Time to First Token----------------
Mean TTFT (ms):                          4731.56
Median TTFT (ms):                        2442.61
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           13309.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          36.97
Median TPOT (ms):                        19.08
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           103.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Uzun bağlam profili

Temel arka uç parametreleri:

--reasoning-parser=qwen3
--max-model-len=32768

Temel benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             9
Benchmark duration (s):                  100.89
Total input tokens:                      3200993
Total generated tokens:                  10000
Request throughput (req/s):              0.99
Output token throughput (tok/s):         100.96
Peak output token throughput (tok/s):    198.08
Peak concurrent requests:                9.00
Total Token throughput (tok/s):          32416.84
----------------------Latency---------------------
Mean Latency(s):                          7.88
Median Latency(s):                        8.10
P95 Latency(s):                           N/A
P99 Latency(s):                           8.54
---------------Time to First Token----------------
Mean TTFT (ms):                          1021.12
Median TTFT (ms):                        1020.12
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           1041.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.21
Median TPOT (ms):                        10.20
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           10.42
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Optimize edilmiş arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Optimize edilmiş benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             14
Benchmark duration (s):                  100.13
Total input tokens:                      3200993
Total generated tokens:                  10000
Request throughput (req/s):              1.00
Output token throughput (tok/s):         101.71
Peak output token throughput (tok/s):    227.24
Peak concurrent requests:                14.00
Total Token throughput (tok/s):          32658.73
----------------------Latency---------------------
Mean Latency(s):                          8.00
Median Latency(s):                        8.00
P99 Latency(s):                           12.36
---------------Time to First Token----------------
Mean TTFT (ms):                          997.50
Median TTFT (ms):                        981.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           1421.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.97
Median TPOT (ms):                        9.81
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           14.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================

Yoğun üretim profili

Temel arka uç parametreleri:

--reasoning-parser=qwen3
--max-model-len=32768

Temel benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             18
Benchmark duration (s):                  214.96
Total input tokens:                      201982
Total generated tokens:                  400000
Request throughput (req/s):              0.93
Output token throughput (tok/s):         1975.75
Peak output token throughput (tok/s):    123795.17
Peak concurrent requests:                18.00
Total Token throughput (tok/s):          2973.42
----------------------Latency---------------------
Mean Latency(s):                          17.48
Median Latency(s):                        17.59
P99 Latency(s):                           17.60
---------------Time to First Token----------------
Mean TTFT (ms):                          54.95
Median TTFT (ms):                        55.17
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           59.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.33
Median TPOT (ms):                        0.03
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           8.79
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.31
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            8.77
==================================================

Optimize edilmiş arka uç parametreleri:

--reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--max-num-seqs=512

Optimize edilmiş benchmark sonucu

============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             18
Benchmark duration (s):                  214.98
Total input tokens:                      201982
Total generated tokens:                  400000
Request throughput (req/s):              0.93
Output token throughput (tok/s):         2010.10
Peak output token throughput (tok/s):    59750.47
Peak concurrent requests:                18.00
Total Token throughput (tok/s):          3025.11
----------------------Latency---------------------
Mean Latency(s):                          17.55
Median Latency(s):                        17.66
P99 Latency(s):                           17.67
---------------Time to First Token----------------
Mean TTFT (ms):                          55.85
Median TTFT (ms):                        56.01
P95 TTFT (ms):                           N/A
P99 TTFT (ms):                           60.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.12
Median TPOT (ms):                        0.03
P95 TPOT (ms):                           N/A
P99 TPOT (ms):                           0.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.09
Median ITL (ms):                         0.00
P95 ITL (ms):                            N/A
P99 ITL (ms):                            0.00
==================================================