コンテンツにスキップ

Qwen3.5-35B-A3B のスループット最適化

結論

スループット最適化の結果

NVIDIA H200 上で Qwen3.5-35B-A3B のスループットを最適化するための推奨設定：

推論サービスのコマンド 
1
2
3
4
5
vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \
    --reasoning-parser=qwen3 \
    --performance-mode=throughput \
    --max-model-len=32768 \
    --enable-prefix-caching

最適化前後のベンチマーク結果の比較：

ベンチマークケース 基準設定（最適化なしの vLLM） 最適化後
ShareGPT プロファイル 合計 TPS: 9632.01
平均 TPOT(ms): 25.89		 合計 TPS: 10570.88 (+9.75%)
平均 TPOT(ms): 24.71
スループット重視プロファイル 合計 TPS: 37934.72
平均 TPOT(ms): 45.39		 合計 TPS: 50464.84 (+33.03%)
平均 TPOT(ms): 41.12
長コンテキスト プロファイル 合計 TPS: 44993.20
平均 TPOT(ms): 319.79		 合計 TPS: 56424.42 (+25.41%)
平均 TPOT(ms): 227.43
生成の多いワークロード プロファイル 合計 TPS: 10455.38
平均 TPOT(ms): 2.48		 合計 TPS: 12258.79 (+17.25%)
平均 TPOT(ms): 2.92

注記

  1. 本ベンチマークは、考えられるすべての最適化の組み合わせを網羅していません。たとえば、デフォルト設定で最も性能が高い推論エンジンを選び、それを追加調整の出発点としています。このような候補の絞り込みで得られるのは局所最適解であり、大域最適解とは限りません。
  2. 最大バッチサイズ、スケジューリング設定、拡張 KV キャッシュ、CUDA graph など、利用シナリオに応じた最適化もあります。本書の結論は、用途に合わせてさらに最適化するための出発点として利用できます。
  3. 本テストは特定のハードウェアとソフトウェアの構成で実施しています。推論エンジンの進化により、結論が変わる可能性があります。
  4. 量子化は精度に影響する場合がありますが、多くのモデルでは FP8 量子化による精度低下は 1% 未満です。詳細は評価結果を参照してください。そのため、低レイテンシーでのモデル提供には FP8 量子化を強く推奨します。
  5. 投機的デコーディングは、低並列度のリクエストのレイテンシーを大きく短縮できます。ただし、ベンチマークデータセットの分布やドラフトモデルの選択によって効果が異なります。たとえば、ここで選んだドラフトモデルは英語データで学習されているため、他の言語では十分な性能が得られない場合があります。

記載漏れや最新の変更を反映すべき点があれば、お知らせください。

実験環境

モデル

  • Qwen/Qwen3.5-35B-A3B

ハードウェア

NVIDIA H200

エンジンのバージョン

  • vLLM v0.17.1
  • SGLang v0.5.9

ベンチマーク手法

本プロジェクトでは、推論ワークロードに GPUStack のワンクリックベンチマーク機能を使用します。本書のベンチマークテストも、この手順で実行しました。

GPUStack のベンチマークは guidellm を基盤とし、ラッパープロジェクト benchmark-runner を介して実装されています。

以下に示すベンチマーク構成について、GPUStack がモデルのデプロイ、ベンチマークジョブの送信、結果の収集を処理します。

ベンチマークプロファイル

ShareGPT
1
2
3
dataset_name: ShareGPT
request_rate: 1000
total_requests: 1000
スループット
1
2
3
4
5
6
dataset_name: Random
dataset_input_tokens: 1024
dataset_output_tokens: 128
dataset_seed: 42
request_rate: 1000
total_requests: 1000
長コンテキスト
1
2
3
4
5
6
dataset_name: Random
dataset_input_tokens: 32000
dataset_output_tokens: 100
dataset_seed: 42
request_rate: 1000
total_requests: 100
生成の多いワークロード
1
2
3
4
5
6
dataset_name: Random
dataset_input_tokens: 1000
dataset_output_tokens: 2000
dataset_seed: 42
request_rate: 1000
total_requests: 200

オープンソースツールによる代替手順

GPUStack を使用しない場合は、GPUStack のベンチマーク手順を直接 guidellm benchmark コマンドで置き換えることができます。

dataset_name: ShareGPT を指定したプロファイルの場合：

1
2
3
4
5
6
7
guidellm benchmark \
  --target ${target} \
  --profile constant \
  --rate ${request_rate} \
  --max-requests ${total_requests} \
  --processor ${model_path} \
  --data ./ShareGPT_V3_unfiltered_cleaned_split.json

dataset_name: Random を指定したプロファイルの場合：

1
2
3
4
5
6
7
8
guidellm benchmark \
  --target ${target} \
  --profile constant \
  --rate ${request_rate} \
  --max-requests ${total_requests} \
  --processor ${model_path} \
  --data "prompt_tokens=${dataset_input_tokens},output_tokens=${dataset_output_tokens}" \
  --random-seed ${dataset_seed:-42}

実験結果

推論エンジンの基準設定

vLLM

  • プロファイル： ShareGPT
  • バックエンドパラメーター: 
    1
2
    --reasoning-parser=qwen3
--max-model-len=32768
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  70.72
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              14.14
Output token throughput (tok/s):         4347.55
Peak output token throughput (tok/s):    30810010.86
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          9632.01
----------------------Latency---------------------
Mean Latency(s):                          30.19
Median Latency(s):                        27.48
P95 Latency(s):                           60.73
P99 Latency(s):                           65.37
---------------Time to First Token----------------
Mean TTFT (ms):                          4072.64
Median TTFT (ms):                        5786.23
P95 TTFT (ms):                           9276.23
P99 TTFT (ms):                           10204.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.89
Median TPOT (ms):                        11.63
P95 TPOT (ms):                           119.58
P99 TPOT (ms):                           161.45
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.46
Median ITL (ms):                         0.00
P95 ITL (ms):                            101.72
P99 ITL (ms):                            132.49
==================================================

SGLang

  • プロファイル： ShareGPT
  • バックエンドパラメーター: 
    1
2
    --reasoning-parser=qwen3
--context-length=32768
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  124.99
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              8.00
Output token throughput (tok/s):         2254.86
Peak output token throughput (tok/s):    125470.63
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          4995.66
----------------------Latency---------------------
Mean Latency(s):                          50.86
Median Latency(s):                        53.56
P95 Latency(s):                           82.38
P99 Latency(s):                           95.75
---------------Time to First Token----------------
Mean TTFT (ms):                          34359.37
Median TTFT (ms):                        44424.39
P95 TTFT (ms):                           50796.35
P99 TTFT (ms):                           53280.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          180.73
Median TPOT (ms):                        145.27
P95 TPOT (ms):                           376.50
P99 TPOT (ms):                           743.84
---------------Inter-token Latency----------------
Mean ITL (ms):                           58.84
Median ITL (ms):                         63.82
P95 ITL (ms):                            70.78
P99 ITL (ms):                            73.69
==================================================
  • 要約： vLLM 合計 TPS = 9632.01, SGLang 合計 TPS = 4995.66. vLLM は 4,636.35 tok/s 高速 (92.8%); 平均 TPOT = 25.89 ms 対 180.73 ms, 短縮量 154.84 ms (85.67%).

量子化

  • プロファイル： ShareGPT
  • バックエンドパラメーター: 
    1
2
    --reasoning-parser=qwen3
--max-model-len=32768
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  65.09
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              15.36
Output token throughput (tok/s):         4487.88
Peak output token throughput (tok/s):    22765417.84
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          9942.93
----------------------Latency---------------------
Mean Latency(s):                          28.47
Median Latency(s):                        25.85
P95 Latency(s):                           57.18
P99 Latency(s):                           60.94
---------------Time to First Token----------------
Mean TTFT (ms):                          4477.45
Median TTFT (ms):                        6644.58
P95 TTFT (ms):                           10070.32
P99 TTFT (ms):                           10452.25
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.85
Median TPOT (ms):                        12.27
P95 TPOT (ms):                           105.60
P99 TPOT (ms):                           155.56
---------------Inter-token Latency----------------
Mean ITL (ms):                           8.97
Median ITL (ms):                         0.00
P95 ITL (ms):                            82.98
P99 ITL (ms):                            122.57
==================================================

プレフィックスキャッシュ && 量子化

  • プロファイル： ShareGPT
  • バックエンドパラメーター: 
    1
2
3
    --reasoning-parser=qwen3
--enable-prefix-caching
--max-model-len=32768
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  66.14
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              15.12
Output token throughput (tok/s):         4666.64
Peak output token throughput (tok/s):    33515595.85
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          10338.96
----------------------Latency---------------------
Mean Latency(s):                          28.95
Median Latency(s):                        26.27
P95 Latency(s):                           58.15
P99 Latency(s):                           61.94
---------------Time to First Token----------------
Mean TTFT (ms):                          4274.88
Median TTFT (ms):                        6301.11
P95 TTFT (ms):                           9633.89
P99 TTFT (ms):                           10096.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.69
Median TPOT (ms):                        11.84
P95 TPOT (ms):                           107.47
P99 TPOT (ms):                           157.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.53
Median ITL (ms):                         0.00
P95 ITL (ms):                            85.45
P99 ITL (ms):                            126.18
==================================================

パフォーマンスモード && プレフィックスキャッシュ && 量子化

  • プロファイル： ShareGPT
  • バックエンドパラメーター: 
    1
2
3
4
    --reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--enable-prefix-caching
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  65.25
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              15.33
Output token throughput (tok/s):         4771.31
Peak output token throughput (tok/s):    41235027.15
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          10570.88
----------------------Latency---------------------
Mean Latency(s):                          28.45
Median Latency(s):                        25.64
P95 Latency(s):                           57.14
P99 Latency(s):                           60.99
---------------Time to First Token----------------
Mean TTFT (ms):                          4123.08
Median TTFT (ms):                        6272.83
P95 TTFT (ms):                           8823.87
P99 TTFT (ms):                           9032.36
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.71
Median TPOT (ms):                        11.49
P95 TPOT (ms):                           108.44
P99 TPOT (ms):                           153.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.10
Median ITL (ms):                         0.00
P95 ITL (ms):                            84.74
P99 ITL (ms):                            124.30
==================================================

KV キャッシュのデータ型 && プレフィックスキャッシュ && パフォーマンスモード && 量子化

  • プロファイル： ShareGPT
  • バックエンドパラメーター: 
    1
2
3
4
5
    --reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--enable-prefix-caching
--kv-cache-dtype=fp8
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  66.34
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              15.07
Output token throughput (tok/s):         4677.83
Peak output token throughput (tok/s):    34141109.18
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          10363.77
----------------------Latency---------------------
Mean Latency(s):                          28.91
Median Latency(s):                        26.27
P95 Latency(s):                           58.08
P99 Latency(s):                           62.05
---------------Time to First Token----------------
Mean TTFT (ms):                          4166.57
Median TTFT (ms):                        6373.29
P95 TTFT (ms):                           8930.75
P99 TTFT (ms):                           9126.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.91
Median TPOT (ms):                        11.41
P95 TPOT (ms):                           111.86
P99 TPOT (ms):                           156.62
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.14
Median ITL (ms):                         0.00
P95 ITL (ms):                            89.13
P99 ITL (ms):                            125.58
==================================================

投機的デコーディング && 量子化

  • プロファイル： ShareGPT
  • バックエンドパラメーター: 
    1
2
3
    --reasoning-parser=qwen3
--speculative-config={"method":"mtp","num_speculative_tokens":4}
--max-model-len=32768
ベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     516
Maximum request concurrency:             512
Benchmark duration (s):                  17.44
Total input tokens:                      54307
Total generated tokens:                  112466
Request throughput (req/s):              29.59
Output token throughput (tok/s):         10462.21
Peak output token throughput (tok/s):    19149104.07
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          15514.15
----------------------Latency---------------------
Mean Latency(s):                          16.92
Median Latency(s):                        17.05
P95 Latency(s):                           17.28
P99 Latency(s):                           17.30
---------------Time to First Token----------------
Mean TTFT (ms):                          1956.19
Median TTFT (ms):                        0.00
P95 TTFT (ms):                           7744.57
P99 TTFT (ms):                           8032.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          66.41
Median TPOT (ms):                        44.74
P95 TPOT (ms):                           149.10
P99 TPOT (ms):                           477.78
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            0.00
P99 ITL (ms):                            0.00
==================================================

最適化項目の一覧

ベンチマークケース 最適化後 基準設定
量子化 合計 TPS: 9942.93 (+3.23%)
平均 TPOT(ms): 24.85		 合計 TPS: 9632.01
平均 TPOT(ms): 25.89
プレフィックスキャッシュ && 量子化 合計 TPS: 10338.96 (+7.34%)
平均 TPOT(ms): 24.69		 合計 TPS: 9632.01
平均 TPOT(ms): 25.89
パフォーマンスモード && プレフィックスキャッシュ && 量子化 合計 TPS: 10570.88 (+9.75%)
平均 TPOT(ms): 24.71		 合計 TPS: 9632.01
平均 TPOT(ms): 25.89
KV キャッシュのデータ型 && パフォーマンスモード && プレフィックスキャッシュ && 量子化 合計 TPS: 10363.77 (+7.60%,前の +9.75% を下回るため最終コマンドでは採用せず)
平均 TPOT(ms): 24.91		 合計 TPS: 9632.01
平均 TPOT(ms): 25.89
投機的デコーディング && 量子化 合計 TPS: 15514.15 (成功率： 51.6%, 最適化を不採用)
平均 TPOT(ms): 66.41		 合計 TPS: 9632.01
平均 TPOT(ms): 25.89

その他のベンチマークケース

ShareGPT

  • 基準設定のバックエンドパラメーター: 
    1
2
    --reasoning-parser=qwen3
--max-model-len=32768
基準設定のベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  70.72
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              14.14
Output token throughput (tok/s):         4347.55
Peak output token throughput (tok/s):    30810010.86
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          9632.01
----------------------Latency---------------------
Mean Latency(s):                          30.19
Median Latency(s):                        27.48
P95 Latency(s):                           60.73
P99 Latency(s):                           65.37
---------------Time to First Token----------------
Mean TTFT (ms):                          4072.64
Median TTFT (ms):                        5786.23
P95 TTFT (ms):                           9276.23
P99 TTFT (ms):                           10204.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.89
Median TPOT (ms):                        11.63
P95 TPOT (ms):                           119.58
P99 TPOT (ms):                           161.45
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.46
Median ITL (ms):                         0.00
P95 ITL (ms):                            101.72
P99 ITL (ms):                            132.49
==================================================
  • 最適化後のバックエンドパラメーター: 
    1
2
3
4
    --reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--enable-prefix-caching
最適化後のベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  65.25
Total input tokens:                      342058
Total generated tokens:                  281412
Request throughput (req/s):              15.33
Output token throughput (tok/s):         4771.31
Peak output token throughput (tok/s):    41235027.15
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          10570.88
----------------------Latency---------------------
Mean Latency(s):                          28.45
Median Latency(s):                        25.64
P95 Latency(s):                           57.14
P99 Latency(s):                           60.99
---------------Time to First Token----------------
Mean TTFT (ms):                          4123.08
Median TTFT (ms):                        6272.83
P95 TTFT (ms):                           8823.87
P99 TTFT (ms):                           9032.36
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.71
Median TPOT (ms):                        11.49
P95 TPOT (ms):                           108.44
P99 TPOT (ms):                           153.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.10
Median ITL (ms):                         0.00
P95 ITL (ms):                            84.74
P99 ITL (ms):                            124.30
==================================================

スループット用プロファイル

  • 基準設定のバックエンドパラメーター: 
    1
2
    --reasoning-parser=qwen3
--max-model-len=32768
基準設定のベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  40.37
Total input tokens:                      1033932
Total generated tokens:                  128000
Request throughput (req/s):              24.77
Output token throughput (tok/s):         4178.94
Peak output token throughput (tok/s):    4782814.36
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          37934.72
----------------------Latency---------------------
Mean Latency(s):                          19.96
Median Latency(s):                        20.63
P95 Latency(s):                           32.86
P99 Latency(s):                           34.01
---------------Time to First Token----------------
Mean TTFT (ms):                          5809.82
Median TTFT (ms):                        5564.25
P95 TTFT (ms):                           14821.97
P99 TTFT (ms):                           16424.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          45.39
Median TPOT (ms):                        43.47
P95 TPOT (ms):                           115.80
P99 TPOT (ms):                           128.32
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            0.00
P99 ITL (ms):                            0.00
==================================================
  • 最適化後のバックエンドパラメーター: 
    1
2
3
4
    --reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--enable-prefix-caching
最適化後のベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             512
Benchmark duration (s):                  33.09
Total input tokens:                      1033932
Total generated tokens:                  128000
Request throughput (req/s):              30.04
Output token throughput (tok/s):         5559.27
Peak output token throughput (tok/s):    8426225.08
Peak concurrent requests:                512.00
Total Token throughput (tok/s):          50464.84
----------------------Latency---------------------
Mean Latency(s):                          16.41
Median Latency(s):                        18.57
P95 Latency(s):                           25.97
P99 Latency(s):                           26.61
---------------Time to First Token----------------
Mean TTFT (ms):                          5263.31
Median TTFT (ms):                        5797.34
P95 TTFT (ms):                           12306.78
P99 TTFT (ms):                           13313.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          41.12
Median TPOT (ms):                        45.29
P95 TPOT (ms):                           96.15
P99 TPOT (ms):                           104.01
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            0.00
P99 ITL (ms):                            0.00
==================================================

長コンテキスト用プロファイル

  • 基準設定のバックエンドパラメーター: 
    1
2
    --reasoning-parser=qwen3
--max-model-len=32768
基準設定のベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             100
Benchmark duration (s):                  73.72
Total input tokens:                      3200993
Total generated tokens:                  10000
Request throughput (req/s):              1.36
Output token throughput (tok/s):         140.12
Peak output token throughput (tok/s):    309.07
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          44993.20
----------------------Latency---------------------
Mean Latency(s):                          47.26
Median Latency(s):                        48.95
P95 Latency(s):                           61.84
P99 Latency(s):                           62.18
---------------Time to First Token----------------
Mean TTFT (ms):                          31979.13
Median TTFT (ms):                        31950.33
P95 TTFT (ms):                           55557.79
P99 TTFT (ms):                           57562.19
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          319.79
Median TPOT (ms):                        319.50
P95 TPOT (ms):                           555.58
P99 TPOT (ms):                           575.62
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            0.00
P99 ITL (ms):                            0.00
==================================================
  • 最適化後のバックエンドパラメーター: 
    1
2
3
4
    --reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--enable-prefix-caching
最適化後のベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             100
Benchmark duration (s):                  59.32
Total input tokens:                      3200993
Total generated tokens:                  10000
Request throughput (req/s):              1.69
Output token throughput (tok/s):         175.72
Peak output token throughput (tok/s):    950.60
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          56424.42
----------------------Latency---------------------
Mean Latency(s):                          44.56
Median Latency(s):                        46.68
P95 Latency(s):                           51.00
P99 Latency(s):                           51.33
---------------Time to First Token----------------
Mean TTFT (ms):                          22743.47
Median TTFT (ms):                        22414.07
P95 TTFT (ms):                           42138.77
P99 TTFT (ms):                           43901.20
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          227.43
Median TPOT (ms):                        224.14
P95 TPOT (ms):                           421.39
P99 TPOT (ms):                           439.01
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            0.00
P99 ITL (ms):                            0.00
==================================================

生成の多いワークロード用プロファイル

  • 基準設定のバックエンドパラメーター: 
    1
2
    --reasoning-parser=qwen3
--max-model-len=32768
基準設定のベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             200
Benchmark duration (s):                  57.74
Total input tokens:                      201982
Total generated tokens:                  400000
Request throughput (req/s):              3.45
Output token throughput (tok/s):         6947.30
Peak output token throughput (tok/s):    77961040.89
Peak concurrent requests:                200.00
Total Token throughput (tok/s):          10455.38
----------------------Latency---------------------
Mean Latency(s):                          57.43
Median Latency(s):                        57.45
P95 Latency(s):                           57.54
P99 Latency(s):                           57.57
---------------Time to First Token----------------
Mean TTFT (ms):                          1915.55
Median TTFT (ms):                        1956.25
P95 TTFT (ms):                           3537.91
P99 TTFT (ms):                           3608.07
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.48
Median TPOT (ms):                        1.00
P95 TPOT (ms):                           28.70
P99 TPOT (ms):                           28.76
---------------Inter-token Latency----------------
Mean ITL (ms):                           1.52
Median ITL (ms):                         0.00
P95 ITL (ms):                            26.98
P99 ITL (ms):                            27.96
==================================================
  • 最適化後のバックエンドパラメーター: 
    1
2
3
4
    --reasoning-parser=qwen3
--performance-mode=throughput
--max-model-len=32768
--enable-prefix-caching
最適化後のベンチマーク結果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
============ Serving Benchmark Result ============
Successful requests:                     200
Maximum request concurrency:             200
Benchmark duration (s):                  49.34
Total input tokens:                      201982
Total generated tokens:                  400000
Request throughput (req/s):              4.05
Output token throughput (tok/s):         8145.62
Peak output token throughput (tok/s):    100865026.05
Peak concurrent requests:                200.00
Total Token throughput (tok/s):          12258.79
----------------------Latency---------------------
Mean Latency(s):                          49.16
Median Latency(s):                        49.16
P95 Latency(s):                           49.18
P99 Latency(s):                           49.20
---------------Time to First Token----------------
Mean TTFT (ms):                          1574.69
Median TTFT (ms):                        1624.98
P95 TTFT (ms):                           2718.49
P99 TTFT (ms):                           2763.35
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.92
Median TPOT (ms):                        0.82
P95 TPOT (ms):                           24.58
P99 TPOT (ms):                           24.59
---------------Inter-token Latency----------------
Mean ITL (ms):                           2.14
Median ITL (ms):                         0.00
P95 ITL (ms):                            23.67
P99 ITL (ms):                            23.99
==================================================