Ana içeriğe geç

NVIDIA H100 GPU üzerinde GPT-OSS-120B işlem hacmi optimizasyonu

Sonuç

gpt-oss-120b-h100

H100 GPU üzerinde GPT-OSS-120B işlem hacmini optimize etmek için önerilen yapılandırma:

Model sunum komutu 
1
vllm serve openai/gpt-oss-120b --async-scheduling -tp 2

Optimizasyon öncesi ve sonrası benchmark sonuçlarının karşılaştırması:

Benchmark senaryosu temel yapılandırma (optimizasyonsuz vLLM) Optimize edilmiş
ShareGPT Toplam TPS: 6095.37
Ortalama TPOT(ms): 54.10		 Toplam TPS: 16042.19 (+31.6%/GPU)
Ortalama TPOT(ms): 86.54
Kısa istem Toplam TPS: 17059.15
Ortalama TPOT(ms): 304.80		 Toplam TPS: 31517.47 (-7.6%/GPU)
Ortalama TPOT(ms): 183.05
Orta uzunlukta istem Toplam TPS: 14385.69
Ortalama TPOT(ms): 41.35		 Toplam TPS: 34083.02 (+18.5%/GPU)
Ortalama TPOT(ms): 139.32
Uzun istem Toplam TPS: 14247.47
Ortalama TPOT(ms): 31.70		 Toplam TPS: 34491.13 (+21.1%/GPU)
Ortalama TPOT(ms): 164.73
Çok uzun istem Toplam TPS: 16136.57
Ortalama TPOT(ms): 23.99		 Toplam TPS: 32471.01 (+0.6%/GPU)
Ortalama TPOT(ms): 220.32

Not

  1. Benchmark testlerimiz olası tüm optimizasyon birleşimlerini kapsamaz. Örneğin, varsayılan yapılandırmasında en iyi performansı gösteren çıkarım motorunu sonraki ayarlamalar için başlangıç noktası olarak seçeriz. Bu eleme yaklaşımı yerel bir optimum verir; bu, genel optimum olmayabilir.
  2. En büyük toplu iş boyutu, zamanlama yapılandırması, genişletilmiş KV önbelleği, CUDA graph, Torch Compile gibi belirli kullanım senaryolarına bağlı başka optimizasyonlar da vardır. Bu belgedeki sonuçlar, daha hedefli optimizasyonlar için başlangıç noktası olabilir.
  3. Testler belirli donanım ve yazılım yapılandırmalarında yürütülmüştür. Çıkarım motorlarındaki gelişmeler sonuçları değiştirebilir.

Eksik noktalar veya yeni değişiklikleri yansıtacak güncellemeler varsa bize bildirin.

Optimizasyon hedefi

Yüksek eş zamanlı isteklerde yüksek işlem hacmi sağlar.

Deney kurulumu

Model

GPT-OSS-120B

Donanım

Tek düğümde 2 × NVIDIA H100 SXM GPU.

Motor sürümü

  • vLLM: v0.10.2
  • SGLang: v0.5.3rc0
  • TensorRT-LLM: v1.0.0

Benchmark veri kümesi

  1. ShareGPT
  2. Farklı dizi uzunluklarına sahip rastgele veri kümesi:
    • Çok uzun istem: 32000 giriş token'ı, 100 çıkış token'ı
    • Uzun istem: 4000 giriş token'ı, 200 çıkış token'ı
    • Orta uzunlukta istem: 2000 giriş token'ı, 100 çıkış token'ı
    • Kısa istem: 128 giriş token'ı, 4 çıkış token'ı

Benchmark betiği

Model performansını ölçmek için vLLM bench CLI aracını kullanıyoruz. Benchmark şu komutla çalıştırılır:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model openai/gpt-oss-120b --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model openai/gpt-oss-120b --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Deney sonuçları

1. Çıkarım motoru seçimi

vLLM

Model sunum betiği 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  67.40
Total input tokens:                      215312
Total generated tokens:                  195544
Request throughput (req/s):              14.84
Output token throughput (tok/s):         2901.05
Total Token throughput (tok/s):          6095.37
---------------Time to First Token----------------
Mean TTFT (ms):                          24830.45
Median TTFT (ms):                        25973.43
P99 TTFT (ms):                           54033.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          54.10
Median TPOT (ms):                        47.57
P99 TPOT (ms):                           193.25
---------------Inter-token Latency----------------
Mean ITL (ms):                           46.65
Median ITL (ms):                         37.98
P99 ITL (ms):                            121.49
==================================================

SGLang

Model sunum betiği 
1
python3 -m sglang.launch_server --model-path openai/gpt-oss-120b --host 0.0.0.0 --port 8000
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  227.00
Total input tokens:                      215312
Total generated tokens:                  196540
Request throughput (req/s):              4.41
Output token throughput (tok/s):         865.81
Total Token throughput (tok/s):          1814.32
---------------Time to First Token----------------
Mean TTFT (ms):                          132612.71
Median TTFT (ms):                        139831.95
P99 TTFT (ms):                           216875.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          57.34
Median TPOT (ms):                        39.11
P99 TPOT (ms):                           345.30
---------------Inter-token Latency----------------
Mean ITL (ms):                           51.23
Median ITL (ms):                         21.25
P99 ITL (ms):                            1328.36
==================================================

TensorRT-LLM

Model sunum betiği 
1
trtllm-serve openai/gpt-oss-120b --max_seq_len 32768
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  191.25
Total input tokens:                      215312
Total generated tokens:                  193874
Request throughput (req/s):              5.23
Output token throughput (tok/s):         1013.72
Total Token throughput (tok/s):          2139.54
---------------Time to First Token----------------
Mean TTFT (ms):                          55896.64
Median TTFT (ms):                        49891.09
P99 TTFT (ms):                           138440.71
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          233.20
Median TPOT (ms):                        213.25
P99 TPOT (ms):                           790.11
---------------Inter-token Latency----------------
Mean ITL (ms):                           101.13
Median ITL (ms):                         53.81
P99 ITL (ms):                            389.65
==================================================

Sonuç: vLLM (6095.37 tok/s) > TensorRT-LLM (2139.54 tok/s) > SGLang(1814.32 tok/s)

2. vLLM için Asenkron zamanlama

Model sunum betiği 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  63.65
Total input tokens:                      215312
Total generated tokens:                  187289
Request throughput (req/s):              15.71
Output token throughput (tok/s):         2942.57
Total Token throughput (tok/s):          6325.41
---------------Time to First Token----------------
Mean TTFT (ms):                          23629.95
Median TTFT (ms):                        24595.25
P99 TTFT (ms):                           51018.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          52.66
Median TPOT (ms):                        46.51
P99 TPOT (ms):                           182.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           45.31
Median ITL (ms):                         36.40
P99 ITL (ms):                            111.51
==================================================

Sonuç: asenkron zamanlama etkinleştirildiğinde işlem hacmi 6095.37 tok/s değerinden 6325.41 tok/s değerine yükselmiştir.

3. vLLM için Paralellik

TP2

Model sunum betiği 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling -tp 2
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  25.65
Total input tokens:                      215312
Total generated tokens:                  196205
Request throughput (req/s):              38.98
Output token throughput (tok/s):         7648.67
Total Token throughput (tok/s):          16042.19
---------------Time to First Token----------------
Mean TTFT (ms):                          4612.68
Median TTFT (ms):                        4600.39
P99 TTFT (ms):                           7587.07
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          86.54
Median TPOT (ms):                        51.57
P99 TPOT (ms):                           215.21
---------------Inter-token Latency----------------
Mean ITL (ms):                           43.02
Median ITL (ms):                         31.37
P99 ITL (ms):                            223.98
==================================================

TP4

Model sunum betiği 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling -tp 4
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  17.32
Total input tokens:                      215312
Total generated tokens:                  196174
Request throughput (req/s):              57.74
Output token throughput (tok/s):         11327.22
Total Token throughput (tok/s):          23759.49
---------------Time to First Token----------------
Mean TTFT (ms):                          2952.04
Median TTFT (ms):                        2871.45
P99 TTFT (ms):                           4867.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          55.77
Median TPOT (ms):                        34.49
P99 TPOT (ms):                           136.33
---------------Inter-token Latency----------------
Mean ITL (ms):                           28.57
Median ITL (ms):                         21.75
P99 ITL (ms):                            149.71
==================================================

PP2

Model sunum betiği 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling -pp 2
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  36.74
Total input tokens:                      215312
Total generated tokens:                  196089
Request throughput (req/s):              27.22
Output token throughput (tok/s):         5337.65
Total Token throughput (tok/s):          11198.56
---------------Time to First Token----------------
Mean TTFT (ms):                          4471.76
Median TTFT (ms):                        4392.85
P99 TTFT (ms):                           6960.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          115.37
Median TPOT (ms):                        69.62
P99 TPOT (ms):                           371.01
---------------Inter-token Latency----------------
Mean ITL (ms):                           60.87
Median ITL (ms):                         50.83
P99 ITL (ms):                            374.02
==================================================

Sonuç: TP2 işlem hacmini 6325.41 tok/s değerinden 16042.19 tok/s değerine çıkarır; GPU başına token işlem hacmi 26.8% artar.

4. vLLM için Dikkat arka ucu

Varsayılan FlashAttention'dır.

FlashInfer

Model sunum betiği 
1
VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve openai/gpt-oss-120b --max-model-len 32768
Benchmark sonucu 
1
RuntimeError: Worker failed with error 'FlashInfer backend currently does not support attention sinks, please use trtllm on blackwell or flash attention on earlier GPUs.', please check the stack trace above for the root cause

XFormers

Model sunum betiği 
1
VLLM_ATTENTION_BACKEND=XFORMERS vllm serve openai/gpt-oss-120b --max-model-len 32768
Benchmark sonucu 
1
TypeError: XFormersImpl.__init__() got an unexpected keyword argument 'sinks'

5. vLLM için Toplu işteki en fazla token sayısı

Model sunum betiği 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling --max-num-batched-tokens 16384
Benchmark sonucu 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  65.28
Total input tokens:                      215312
Total generated tokens:                  187636
Request throughput (req/s):              15.32
Output token throughput (tok/s):         2874.47
Total Token throughput (tok/s):          6172.93
---------------Time to First Token----------------
Mean TTFT (ms):                          23977.21
Median TTFT (ms):                        24694.18
P99 TTFT (ms):                           51339.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          51.18
Median TPOT (ms):                        45.25
P99 TPOT (ms):                           164.63
---------------Inter-token Latency----------------
Mean ITL (ms):                           44.26
Median ITL (ms):                         35.89
P99 ITL (ms):                            109.05
==================================================

Sonuç: max-num-batched-tokens artırıldığında işlem hacmi biraz azalmıştır.

Optimizasyon seçeneklerinin özeti

Optimizasyon seçeneği İşlem hacmi iyileşmesi
Motor seçimi -
Asenkron zamanlama +3.8%
Paralellik +26.8%
Dikkat arka ucu -
Toplu işteki en fazla token sayısı -

Diğer benchmark senaryoları

Optimize edilmiş yapılandırmanın farklı iş yüklerine genellenebilirliğini değerlendirmek için ek benchmark testleri yaptık.

Temel model sunum betiği 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768
Temel benchmark sonuçları 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  198.92
Total input tokens:                      3200000
Total generated tokens:                  9877
Request throughput (req/s):              0.50
Output token throughput (tok/s):         49.65
Total Token throughput (tok/s):          16136.57
---------------Time to First Token----------------
Mean TTFT (ms):                          99444.04
Median TTFT (ms):                        99439.56
P99 TTFT (ms):                           196214.22
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          23.99
Median TPOT (ms):                        23.69
P99 TPOT (ms):                           31.32
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.13
Median ITL (ms):                         7.26
P99 ITL (ms):                            454.68
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  145.04
Total input tokens:                      1999249
Total generated tokens:                  67205
Request throughput (req/s):              3.45
Output token throughput (tok/s):         463.35
Total Token throughput (tok/s):          14247.47
---------------Time to First Token----------------
Mean TTFT (ms):                          72954.68
Median TTFT (ms):                        73023.24
P99 TTFT (ms):                           141744.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          31.70
Median TPOT (ms):                        32.02
P99 TPOT (ms):                           37.43
---------------Inter-token Latency----------------
Mean ITL (ms):                           33.25
Median ITL (ms):                         12.37
P99 ITL (ms):                            191.40
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  72.87
Total input tokens:                      999120
Total generated tokens:                  49197
Request throughput (req/s):              6.86
Output token throughput (tok/s):         675.11
Total Token throughput (tok/s):          14385.69
---------------Time to First Token----------------
Mean TTFT (ms):                          36234.76
Median TTFT (ms):                        35662.58
P99 TTFT (ms):                           70634.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          41.35
Median TPOT (ms):                        42.86
P99 TPOT (ms):                           48.36
---------------Inter-token Latency----------------
Mean ITL (ms):                           43.61
Median ITL (ms):                         14.92
P99 ITL (ms):                            309.43
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  7.72
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              129.48
Output token throughput (tok/s):         517.91
Total Token throughput (tok/s):          17059.15
---------------Time to First Token----------------
Mean TTFT (ms):                          4850.46
Median TTFT (ms):                        4987.41
P99 TTFT (ms):                           7372.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          304.80
Median TPOT (ms):                        341.94
P99 TPOT (ms):                           360.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           467.00
Median ITL (ms):                         352.58
P99 ITL (ms):                            728.75
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  47.88
Total input tokens:                      22946
Total generated tokens:                  21691
Request throughput (req/s):              2.09
Output token throughput (tok/s):         453.04
Total Token throughput (tok/s):          932.29
---------------Time to First Token----------------
Mean TTFT (ms):                          49.00
Median TTFT (ms):                        47.49
P99 TTFT (ms):                           107.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.31
Median TPOT (ms):                        8.24
P99 TPOT (ms):                           10.12
---------------Inter-token Latency----------------
Mean ITL (ms):                           8.40
Median ITL (ms):                         7.99
P99 ITL (ms):                            23.32
==================================================
Optimize edilmiş model sunum betiği 
1
vllm serve openai/gpt-oss-120b --async-scheduling -tp 2
Optimize edilmiş benchmark sonuçları 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  98.86
Total input tokens:                      3200000
Total generated tokens:                  9955
Request throughput (req/s):              1.01
Output token throughput (tok/s):         100.70
Total Token throughput (tok/s):          32471.01
---------------Time to First Token----------------
Mean TTFT (ms):                          48826.05
Median TTFT (ms):                        48671.00
P99 TTFT (ms):                           96651.74
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          220.32
Median TPOT (ms):                        252.27
P99 TPOT (ms):                           252.79
---------------Inter-token Latency----------------
Mean ITL (ms):                           228.81
Median ITL (ms):                         246.94
P99 ITL (ms):                            481.43
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  59.94
Total input tokens:                      1999249
Total generated tokens:                  68046
Request throughput (req/s):              8.34
Output token throughput (tok/s):         1135.29
Total Token throughput (tok/s):          34491.13
---------------Time to First Token----------------
Mean TTFT (ms):                          29487.34
Median TTFT (ms):                        29235.61
P99 TTFT (ms):                           56201.04
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          164.73
Median TPOT (ms):                        199.03
P99 TPOT (ms):                           220.23
---------------Inter-token Latency----------------
Mean ITL (ms):                           171.11
Median ITL (ms):                         215.59
P99 ITL (ms):                            431.23
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  30.75
Total input tokens:                      999120
Total generated tokens:                  49059
Request throughput (req/s):              16.26
Output token throughput (tok/s):         1595.22
Total Token throughput (tok/s):          34083.02
---------------Time to First Token----------------
Mean TTFT (ms):                          15291.77
Median TTFT (ms):                        15170.31
P99 TTFT (ms):                           28437.70
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          139.32
Median TPOT (ms):                        148.04
P99 TPOT (ms):                           212.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           147.37
Median ITL (ms):                         208.40
P99 ITL (ms):                            421.37
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  4.18
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              239.21
Output token throughput (tok/s):         956.85
Total Token throughput (tok/s):          31517.47
---------------Time to First Token----------------
Mean TTFT (ms):                          2800.15
Median TTFT (ms):                        2412.28
P99 TTFT (ms):                           3856.95
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          183.05
Median TPOT (ms):                        208.66
P99 TPOT (ms):                           232.15
---------------Inter-token Latency----------------
Mean ITL (ms):                           274.85
Median ITL (ms):                         267.83
P99 ITL (ms):                            506.27
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  33.60
Total input tokens:                      22946
Total generated tokens:                  21691
Request throughput (req/s):              2.98
Output token throughput (tok/s):         645.55
Total Token throughput (tok/s):          1328.44
---------------Time to First Token----------------
Mean TTFT (ms):                          37.51
Median TTFT (ms):                        36.68
P99 TTFT (ms):                           58.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.75
Median TPOT (ms):                        5.74
P99 TPOT (ms):                           6.91
---------------Inter-token Latency----------------
Mean ITL (ms):                           5.89
Median ITL (ms):                         5.54
P99 ITL (ms):                            23.43
==================================================