NVIDIA A100 GPU 上での GPT-OSS-120B のスループット最適化

結論

A100 GPU 上で GPT-OSS-120B のスループットを最適化するための推奨設定：

推論サービスのコマンド

vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling --max-num-batched-tokens 4096

最適化前後のベンチマーク結果の比較：

ベンチマークケース	基準設定（最適化なしの vLLM）	最適化後
ShareGPT	合計 TPS: 4768.75 平均 TPOT(ms): 97.57	合計 TPS: 5055.82 (+6.0%) 平均 TPOT(ms): 94.55
短いプロンプト	合計 TPS: 9353.12 平均 TPOT(ms): 182.75	合計 TPS: 10785.60 (+15.3%) 平均 TPOT(ms): 301.66
中程度のプロンプト	合計 TPS: 8456.65 平均 TPOT(ms): 131.94	合計 TPS: 9249.89 (+9.4%) 平均 TPOT(ms): 117.85
長いプロンプト	合計 TPS: 7671.00 平均 TPOT(ms): 112.18	合計 TPS: 8468.21 (+10.4%) 平均 TPOT(ms): 99.77
非常に長いプロンプト	合計 TPS: 4337.21 平均 TPOT(ms): 224.58	合計 TPS: 4605.91 (+6.2%) 平均 TPOT(ms): 201.61

注記

本ベンチマークは、考えられるすべての最適化の組み合わせを網羅していません。たとえば、デフォルト設定で最も性能が高い推論エンジンを選び、それを追加調整の出発点としています。このような候補の絞り込みで得られるのは局所最適解であり、大域最適解とは限りません。
最大バッチサイズ、スケジューリング設定、拡張 KV キャッシュ、CUDA graph、Torch Compile など、利用シナリオに応じた最適化もあります。本書の結論は、用途に合わせてさらに最適化するための出発点として利用できます。
本テストは特定のハードウェアとソフトウェアの構成で実施しています。推論エンジンの進化により、結論が変わる可能性があります。

記載漏れや最新の変更を反映すべき点があれば、お知らせください。

最適化の目標

高並列度のリクエストで高スループットを実現します。

実験環境

モデル

GPT-OSS-120B

ハードウェア

単一ノードに 2 × NVIDIA A100 SXM 80GB GPU。

エンジンのバージョン

vLLM: v0.11.0
SGLang: v0.5.3
TensorRT-LLM: v1.2.0rc0

ベンチマークデータセット

ShareGPT
シーケンス長を変えたランダムデータセット：
- 非常に長いプロンプト： 32000 入力トークン, 100 出力トークン
- 長いプロンプト： 4000 入力トークン, 200 出力トークン
- 中程度のプロンプト： 2000 入力トークン, 100 出力トークン
- 短いプロンプト： 128 入力トークン, 4 出力トークン

ベンチマークスクリプト

モデルの性能評価には vLLM bench CLI ツールを使用します。ベンチマークを実行するコマンドは次のとおりです：

# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model openai/gpt-oss-120b --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model openai/gpt-oss-120b --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

実験結果

1. 推論エンジンの選択

vLLM

推論サービスのスクリプト

vllm serve openai/gpt-oss-120b --max-model-len 32768

ベンチマーク結果

============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  86.39
Total input tokens:                      215312
Total generated tokens:                  196656
Request throughput (req/s):              11.58
Output token throughput (tok/s):         2276.40
Peak output token throughput (tok/s):    3572.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4768.75
---------------Time to First Token----------------
Mean TTFT (ms):                          25918.79
Median TTFT (ms):                        24822.43
P99 TTFT (ms):                           59731.55
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          97.57
Median TPOT (ms):                        90.70
P99 TPOT (ms):                           206.31
---------------Inter-token Latency----------------
Mean ITL (ms):                           86.67
Median ITL (ms):                         70.10
P99 ITL (ms):                            210.76
==================================================

SGLang

推論サービスのスクリプト

python3 -m sglang.launch_server --model-path openai/gpt-oss-120b --host 0.0.0.0 --port 8000 

ベンチマーク結果

============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  597.93
Total input tokens:                      215312
Total generated tokens:                  197133
Request throughput (req/s):              1.67
Output token throughput (tok/s):         329.69
Peak output token throughput (tok/s):    1865.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          689.79
---------------Time to First Token----------------
Mean TTFT (ms):                          275030.66
Median TTFT (ms):                        270126.72
P99 TTFT (ms):                           550929.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          765.78
Median TPOT (ms):                        486.18
P99 TPOT (ms):                           6656.59
---------------Inter-token Latency----------------
Mean ITL (ms):                           458.97
Median ITL (ms):                         112.65
P99 ITL (ms):                            6667.80
==================================================

TensorRT-LLM

推論サービスのスクリプト

trtllm-serve openai/gpt-oss-120b --max_seq_len 32768

ベンチマーク結果

1	`NotImplementedError: WFP4A16 MoE is unsupported on SM80.`

結果： vLLM (4768.75 tok/s) > SGLang(689.79 tok/s) > TensorRT-LLM (サポート対象外)

2. 非同期スケジューリング：vLLM

推論サービスのスクリプト

vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling

ベンチマーク結果

============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  82.35
Total input tokens:                      215312
Total generated tokens:                  195312
Request throughput (req/s):              12.14
Output token throughput (tok/s):         2371.67
Peak output token throughput (tok/s):    3615.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          4986.20
---------------Time to First Token----------------
Mean TTFT (ms):                          24787.19
Median TTFT (ms):                        23575.09
P99 TTFT (ms):                           56631.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          92.70
Median TPOT (ms):                        86.19
P99 TPOT (ms):                           203.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           82.31
Median ITL (ms):                         65.76
P99 ITL (ms):                            205.68
==================================================

結果：非同期スケジューリングを有効にすると、スループットが 4768.75 tok/s から 4986.20 tok/s に向上しました。

3. 並列化：vLLM

TP2

推論サービスのスクリプト

vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling -tp 2

ベンチマーク結果

============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  52.10
Total input tokens:                      215312
Total generated tokens:                  196156
Request throughput (req/s):              19.19
Output token throughput (tok/s):         3765.02
Peak output token throughput (tok/s):    5687.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          7897.72
---------------Time to First Token----------------
Mean TTFT (ms):                          16190.50
Median TTFT (ms):                        15435.84
P99 TTFT (ms):                           36099.41
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          57.98
Median TPOT (ms):                        53.39
P99 TPOT (ms):                           132.88
---------------Inter-token Latency----------------
Mean ITL (ms):                           51.20
Median ITL (ms):                         40.51
P99 ITL (ms):                            134.62
==================================================

4. アテンションバックエンド：vLLM

デフォルトは FlashAttention です。

FlashInfer

推論サービスのスクリプト

VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve openai/gpt-oss-120b --max-model-len 32768

ベンチマーク結果

RuntimeError: Worker failed with error 'FlashInfer backend currently does not support attention sinks, please use trtllm on blackwell or flash attention on earlier GPUs.', please check the stack trace above for the root cause

5. 最大バッチトークン数：vLLM

推論サービスのスクリプト

vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling --max-num-batched-tokens 4096

ベンチマーク結果

# --max-num-batched-tokens 4096
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  81.24
Total input tokens:                      215312
Total generated tokens:                  195400
Request throughput (req/s):              12.31
Output token throughput (tok/s):         2405.35
Peak output token throughput (tok/s):    3767.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5055.82
---------------Time to First Token----------------
Mean TTFT (ms):                          24102.33
Median TTFT (ms):                        23200.31
P99 TTFT (ms):                           55834.89
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          94.55
Median TPOT (ms):                        84.96
P99 TPOT (ms):                           332.44
---------------Inter-token Latency----------------
Mean ITL (ms):                           81.69
Median ITL (ms):                         65.37
P99 ITL (ms):                            263.32
==================================================
# --max-num-batched-tokens 8192
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  81.65
Total input tokens:                      215312
Total generated tokens:                  196734
Request throughput (req/s):              12.25
Output token throughput (tok/s):         2409.38
Peak output token throughput (tok/s):    4073.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5046.28
---------------Time to First Token----------------
Mean TTFT (ms):                          24118.79
Median TTFT (ms):                        22914.78
P99 TTFT (ms):                           56155.47
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          95.68
Median TPOT (ms):                        85.26
P99 TPOT (ms):                           379.93
---------------Inter-token Latency----------------
Mean ITL (ms):                           81.81
Median ITL (ms):                         65.77
P99 ITL (ms):                            232.43
==================================================

結果：max-num-batched-tokens を増やすと、スループットが 4986.20 tok/s から 5046.28 tok/s に向上しました。

最適化項目の一覧

最適化項目	スループット改善率
エンジンの選択	-
非同期スケジューリング	+4.6%
並列化	-
アテンションバックエンド	-
最大バッチトークン数	+1.2%

その他のベンチマークケース

さらに、さまざまなワークロードに対する汎用性を評価するため、最適化済み設定でベンチマークを実施しました。

基準設定の推論スクリプト

vllm serve openai/gpt-oss-120b --max-model-len 32768

基準設定のベンチマーク結果

# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  740.10
Total input tokens:                      3200000
Total generated tokens:                  9988
Request throughput (req/s):              0.14
Output token throughput (tok/s):         13.50
Peak output token throughput (tok/s):    176.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          4337.21
---------------Time to First Token----------------
Mean TTFT (ms):                          368856.64
Median TTFT (ms):                        368713.30
P99 TTFT (ms):                           730947.33
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          224.58
Median TPOT (ms):                        228.19
P99 TPOT (ms):                           231.28
---------------Inter-token Latency----------------
Mean ITL (ms):                           233.21
Median ITL (ms):                         24.23
P99 ITL (ms):                            676.79
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  269.39
Total input tokens:                      1999249
Total generated tokens:                  67259
Request throughput (req/s):              1.86
Output token throughput (tok/s):         249.67
Peak output token throughput (tok/s):    1146.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          7671.00
---------------Time to First Token----------------
Mean TTFT (ms):                          133561.13
Median TTFT (ms):                        134006.62
P99 TTFT (ms):                           263106.36
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          112.18
Median TPOT (ms):                        114.77
P99 TPOT (ms):                           136.26
---------------Inter-token Latency----------------
Mean ITL (ms):                           117.62
Median ITL (ms):                         27.99
P99 ITL (ms):                            353.98
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  123.94
Total input tokens:                      999120
Total generated tokens:                  48981
Request throughput (req/s):              4.03
Output token throughput (tok/s):         395.21
Peak output token throughput (tok/s):    1696.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          8456.65
---------------Time to First Token----------------
Mean TTFT (ms):                          61328.02
Median TTFT (ms):                        61764.97
P99 TTFT (ms):                           120670.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          131.94
Median TPOT (ms):                        138.30
P99 TPOT (ms):                           150.84
---------------Inter-token Latency----------------
Mean ITL (ms):                           139.51
Median ITL (ms):                         214.02
P99 ITL (ms):                            432.26
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  14.09
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              70.99
Output token throughput (tok/s):         283.95
Peak output token throughput (tok/s):    273.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          9353.12
---------------Time to First Token----------------
Mean TTFT (ms):                          7813.18
Median TTFT (ms):                        7750.58
P99 TTFT (ms):                           13935.99
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          182.75
Median TPOT (ms):                        189.53
P99 TPOT (ms):                           193.12
---------------Inter-token Latency----------------
Mean ITL (ms):                           274.25
Median ITL (ms):                         196.02
P99 ITL (ms):                            386.54
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  66.52
Total input tokens:                      22946
Total generated tokens:                  21691
Request throughput (req/s):              1.50
Output token throughput (tok/s):         326.09
Peak output token throughput (tok/s):    368.00
Peak concurrent requests:                11.00
Total Token throughput (tok/s):          671.05
---------------Time to First Token----------------
Mean TTFT (ms):                          61.91
Median TTFT (ms):                        61.55
P99 TTFT (ms):                           112.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.67
Median TPOT (ms):                        11.44
P99 TPOT (ms):                           15.03
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.74
Median ITL (ms):                         11.14
P99 ITL (ms):                            38.75
==================================================  

最適化後の推論スクリプト

vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling --max-num-batched-tokens 4096

最適化後のベンチマーク結果

# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  696.89
Total input tokens:                      3200000
Total generated tokens:                  9828
Request throughput (req/s):              0.14
Output token throughput (tok/s):         14.10
Peak output token throughput (tok/s):    196.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          4605.91
---------------Time to First Token----------------
Mean TTFT (ms):                          346799.49
Median TTFT (ms):                        346737.12
P99 TTFT (ms):                           687309.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          201.61
Median TPOT (ms):                        203.58
P99 TPOT (ms):                           271.48
---------------Inter-token Latency----------------
Mean ITL (ms):                           209.02
Median ITL (ms):                         20.50
P99 ITL (ms):                            1206.69
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  244.13
Total input tokens:                      1999249
Total generated tokens:                  68059
Request throughput (req/s):              2.05
Output token throughput (tok/s):         278.79
Peak output token throughput (tok/s):    1289.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          8468.21
---------------Time to First Token----------------
Mean TTFT (ms):                          121158.11
Median TTFT (ms):                        121158.59
P99 TTFT (ms):                           238837.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          99.77
Median TPOT (ms):                        102.46
P99 TPOT (ms):                           122.61
---------------Inter-token Latency----------------
Mean ITL (ms):                           104.98
Median ITL (ms):                         25.69
P99 ITL (ms):                            397.17
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  113.32
Total input tokens:                      999120
Total generated tokens:                  49103
Request throughput (req/s):              4.41
Output token throughput (tok/s):         433.30
Peak output token throughput (tok/s):    1893.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          9249.89
---------------Time to First Token----------------
Mean TTFT (ms):                          55458.38
Median TTFT (ms):                        56144.82
P99 TTFT (ms):                           110079.18
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          117.85
Median TPOT (ms):                        125.32
P99 TPOT (ms):                           135.13
---------------Inter-token Latency----------------
Mean ITL (ms):                           125.08
Median ITL (ms):                         31.45
P99 ITL (ms):                            575.36
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  12.22
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              81.86
Output token throughput (tok/s):         327.44
Peak output token throughput (tok/s):    360.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10785.60
---------------Time to First Token----------------
Mean TTFT (ms):                          6900.44
Median TTFT (ms):                        6888.59
P99 TTFT (ms):                           11990.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          301.66
Median TPOT (ms):                        320.19
P99 TPOT (ms):                           323.93
---------------Inter-token Latency----------------
Mean ITL (ms):                           452.71
Median ITL (ms):                         327.62
P99 ITL (ms):                            652.15
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  62.98
Total input tokens:                      22946
Total generated tokens:                  21691
Request throughput (req/s):              1.59
Output token throughput (tok/s):         344.39
Peak output token throughput (tok/s):    387.00
Peak concurrent requests:                11.00
Total Token throughput (tok/s):          708.71
---------------Time to First Token----------------
Mean TTFT (ms):                          71.15
Median TTFT (ms):                        70.23
P99 TTFT (ms):                           131.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.03
Median TPOT (ms):                        10.78
P99 TPOT (ms):                           15.48
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.07
Median ITL (ms):                         10.49
P99 ITL (ms):                            36.80
==================================================