跳转至

NVIDIA H100 GPUs 上的 GPT-OSS-120B 吞吐量优化

结论

gpt-oss-120b-h100

GPT-OSS-120B 在 H100 GPUs 上的吞吐量优化推荐配置：

推理服务命令 
1
vllm serve openai/gpt-oss-120b --async-scheduling -tp 2

优化前后的基准测试结果对比：

基准测试用例 基准配置（未做任何优化的 vLLM） 优化后
ShareGPT Total TPS: 6095.37
Mean TPOT(ms): 54.10		 Total TPS: 16042.19 (+31.6%/GPU)
Mean TPOT(ms): 86.54
短提示词 Total TPS: 17059.15
Mean TPOT(ms): 304.80		 Total TPS: 31517.47 (-7.6%/GPU)
Mean TPOT(ms): 183.05
中等长度提示词 Total TPS: 14385.69
Mean TPOT(ms): 41.35		 Total TPS: 34083.02 (+18.5%/GPU)
Mean TPOT(ms): 139.32
长提示词 Total TPS: 14247.47
Mean TPOT(ms): 31.70		 Total TPS: 34491.13 (+21.1%/GPU)
Mean TPOT(ms): 164.73
超长提示词 Total TPS: 16136.57
Mean TPOT(ms): 23.99		 Total TPS: 32471.01 (+0.6%/GPU)
Mean TPOT(ms): 220.32

说明

  1. 基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
  2. 其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph、Torch Compile 等。本文结论可作为进一步针对性优化的起点。
  3. 测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

优化目标

在高并发请求场景下实现高吞吐量。

实验环境

模型

GPT-OSS-120B

硬件

单节点配备 2 × NVIDIA H100 SXM GPU。

引擎版本

  • vLLM: v0.10.2
  • SGLang: v0.5.3rc0
  • TensorRT-LLM: v1.0.0

基准测试数据集

  1. ShareGPT
  2. 不同序列长度的随机数据集：
    • 超长提示词：32000 输入 tokens，100 输出 tokens
    • 长提示词：4000 输入 tokens，200 输出 tokens
    • 中等长度提示词：2000 输入 tokens，100 输出 tokens
    • 短提示词：128 输入 tokens，4 输出 tokens

基准测试脚本

使用 vLLM bench CLI 工具对模型性能进行基准测试。运行基准测试的命令如下：

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model openai/gpt-oss-120b --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model openai/gpt-oss-120b --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

实验结果

1. 选择推理引擎

vLLM

推理服务脚本 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  67.40
Total input tokens:                      215312
Total generated tokens:                  195544
Request throughput (req/s):              14.84
Output token throughput (tok/s):         2901.05
Total Token throughput (tok/s):          6095.37
---------------Time to First Token----------------
Mean TTFT (ms):                          24830.45
Median TTFT (ms):                        25973.43
P99 TTFT (ms):                           54033.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          54.10
Median TPOT (ms):                        47.57
P99 TPOT (ms):                           193.25
---------------Inter-token Latency----------------
Mean ITL (ms):                           46.65
Median ITL (ms):                         37.98
P99 ITL (ms):                            121.49
==================================================

SGLang

推理服务脚本 
1
python3 -m sglang.launch_server --model-path openai/gpt-oss-120b --host 0.0.0.0 --port 8000
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  227.00
Total input tokens:                      215312
Total generated tokens:                  196540
Request throughput (req/s):              4.41
Output token throughput (tok/s):         865.81
Total Token throughput (tok/s):          1814.32
---------------Time to First Token----------------
Mean TTFT (ms):                          132612.71
Median TTFT (ms):                        139831.95
P99 TTFT (ms):                           216875.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          57.34
Median TPOT (ms):                        39.11
P99 TPOT (ms):                           345.30
---------------Inter-token Latency----------------
Mean ITL (ms):                           51.23
Median ITL (ms):                         21.25
P99 ITL (ms):                            1328.36
==================================================

TensorRT-LLM

推理服务脚本 
1
trtllm-serve openai/gpt-oss-120b --max_seq_len 32768
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  191.25
Total input tokens:                      215312
Total generated tokens:                  193874
Request throughput (req/s):              5.23
Output token throughput (tok/s):         1013.72
Total Token throughput (tok/s):          2139.54
---------------Time to First Token----------------
Mean TTFT (ms):                          55896.64
Median TTFT (ms):                        49891.09
P99 TTFT (ms):                           138440.71
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          233.20
Median TPOT (ms):                        213.25
P99 TPOT (ms):                           790.11
---------------Inter-token Latency----------------
Mean ITL (ms):                           101.13
Median ITL (ms):                         53.81
P99 ITL (ms):                            389.65
==================================================

结果：vLLM (6095.37 tok/s) > TensorRT-LLM (2139.54 tok/s) > SGLang(1814.32 tok/s)

2. vLLM 中的 异步调度

推理服务脚本 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  63.65
Total input tokens:                      215312
Total generated tokens:                  187289
Request throughput (req/s):              15.71
Output token throughput (tok/s):         2942.57
Total Token throughput (tok/s):          6325.41
---------------Time to First Token----------------
Mean TTFT (ms):                          23629.95
Median TTFT (ms):                        24595.25
P99 TTFT (ms):                           51018.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          52.66
Median TPOT (ms):                        46.51
P99 TPOT (ms):                           182.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           45.31
Median ITL (ms):                         36.40
P99 ITL (ms):                            111.51
==================================================

结果：启用异步调度后，吞吐量从 6095.37 tok/s 提升至 6325.41 tok/s。

3. vLLM 中的 并行策略

TP2

推理服务脚本 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling -tp 2
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  25.65
Total input tokens:                      215312
Total generated tokens:                  196205
Request throughput (req/s):              38.98
Output token throughput (tok/s):         7648.67
Total Token throughput (tok/s):          16042.19
---------------Time to First Token----------------
Mean TTFT (ms):                          4612.68
Median TTFT (ms):                        4600.39
P99 TTFT (ms):                           7587.07
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          86.54
Median TPOT (ms):                        51.57
P99 TPOT (ms):                           215.21
---------------Inter-token Latency----------------
Mean ITL (ms):                           43.02
Median ITL (ms):                         31.37
P99 ITL (ms):                            223.98
==================================================

TP4

推理服务脚本 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling -tp 4
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  17.32
Total input tokens:                      215312
Total generated tokens:                  196174
Request throughput (req/s):              57.74
Output token throughput (tok/s):         11327.22
Total Token throughput (tok/s):          23759.49
---------------Time to First Token----------------
Mean TTFT (ms):                          2952.04
Median TTFT (ms):                        2871.45
P99 TTFT (ms):                           4867.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          55.77
Median TPOT (ms):                        34.49
P99 TPOT (ms):                           136.33
---------------Inter-token Latency----------------
Mean ITL (ms):                           28.57
Median ITL (ms):                         21.75
P99 ITL (ms):                            149.71
==================================================

PP2

推理服务脚本 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling -pp 2
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  36.74
Total input tokens:                      215312
Total generated tokens:                  196089
Request throughput (req/s):              27.22
Output token throughput (tok/s):         5337.65
Total Token throughput (tok/s):          11198.56
---------------Time to First Token----------------
Mean TTFT (ms):                          4471.76
Median TTFT (ms):                        4392.85
P99 TTFT (ms):                           6960.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          115.37
Median TPOT (ms):                        69.62
P99 TPOT (ms):                           371.01
---------------Inter-token Latency----------------
Mean ITL (ms):                           60.87
Median ITL (ms):                         50.83
P99 ITL (ms):                            374.02
==================================================

结果：TP2 将吞吐量从 6325.41 tok/s 提升至 16042.19 tok/s，每个 GPU 的 token 吞吐量提高 26.8%。

4. vLLM 中的 注意力后端

默认使用 FlashAttention。

FlashInfer

推理服务脚本 
1
VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve openai/gpt-oss-120b --max-model-len 32768
基准测试结果 
1
RuntimeError: Worker failed with error 'FlashInfer backend currently does not support attention sinks, please use trtllm on blackwell or flash attention on earlier GPUs.', please check the stack trace above for the root cause

XFormers

推理服务脚本 
1
VLLM_ATTENTION_BACKEND=XFORMERS vllm serve openai/gpt-oss-120b --max-model-len 32768
基准测试结果 
1
TypeError: XFormersImpl.__init__() got an unexpected keyword argument 'sinks'

5. vLLM 中的 最大批处理 Token 数

推理服务脚本 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768 --async-scheduling --max-num-batched-tokens 16384
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  65.28
Total input tokens:                      215312
Total generated tokens:                  187636
Request throughput (req/s):              15.32
Output token throughput (tok/s):         2874.47
Total Token throughput (tok/s):          6172.93
---------------Time to First Token----------------
Mean TTFT (ms):                          23977.21
Median TTFT (ms):                        24694.18
P99 TTFT (ms):                           51339.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          51.18
Median TPOT (ms):                        45.25
P99 TPOT (ms):                           164.63
---------------Inter-token Latency----------------
Mean ITL (ms):                           44.26
Median ITL (ms):                         35.89
P99 ITL (ms):                            109.05
==================================================

结果：增大 max-num-batched-tokens 后，吞吐量略有下降。

优化选项汇总

优化选项 吞吐量提升
引擎选择 -
异步调度 +3.8%
并行策略 +26.8%
注意力后端 -
最大批处理 Token 数 -

其他基准测试用例

我们进一步对优化后的配置进行了基准测试，评估其在不同工作负载下的泛化表现。

基准配置的推理服务脚本 
1
vllm serve openai/gpt-oss-120b --max-model-len 32768
基准配置的测试结果 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  198.92
Total input tokens:                      3200000
Total generated tokens:                  9877
Request throughput (req/s):              0.50
Output token throughput (tok/s):         49.65
Total Token throughput (tok/s):          16136.57
---------------Time to First Token----------------
Mean TTFT (ms):                          99444.04
Median TTFT (ms):                        99439.56
P99 TTFT (ms):                           196214.22
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          23.99
Median TPOT (ms):                        23.69
P99 TPOT (ms):                           31.32
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.13
Median ITL (ms):                         7.26
P99 ITL (ms):                            454.68
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  145.04
Total input tokens:                      1999249
Total generated tokens:                  67205
Request throughput (req/s):              3.45
Output token throughput (tok/s):         463.35
Total Token throughput (tok/s):          14247.47
---------------Time to First Token----------------
Mean TTFT (ms):                          72954.68
Median TTFT (ms):                        73023.24
P99 TTFT (ms):                           141744.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          31.70
Median TPOT (ms):                        32.02
P99 TPOT (ms):                           37.43
---------------Inter-token Latency----------------
Mean ITL (ms):                           33.25
Median ITL (ms):                         12.37
P99 ITL (ms):                            191.40
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  72.87
Total input tokens:                      999120
Total generated tokens:                  49197
Request throughput (req/s):              6.86
Output token throughput (tok/s):         675.11
Total Token throughput (tok/s):          14385.69
---------------Time to First Token----------------
Mean TTFT (ms):                          36234.76
Median TTFT (ms):                        35662.58
P99 TTFT (ms):                           70634.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          41.35
Median TPOT (ms):                        42.86
P99 TPOT (ms):                           48.36
---------------Inter-token Latency----------------
Mean ITL (ms):                           43.61
Median ITL (ms):                         14.92
P99 ITL (ms):                            309.43
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  7.72
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              129.48
Output token throughput (tok/s):         517.91
Total Token throughput (tok/s):          17059.15
---------------Time to First Token----------------
Mean TTFT (ms):                          4850.46
Median TTFT (ms):                        4987.41
P99 TTFT (ms):                           7372.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          304.80
Median TPOT (ms):                        341.94
P99 TPOT (ms):                           360.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           467.00
Median ITL (ms):                         352.58
P99 ITL (ms):                            728.75
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  47.88
Total input tokens:                      22946
Total generated tokens:                  21691
Request throughput (req/s):              2.09
Output token throughput (tok/s):         453.04
Total Token throughput (tok/s):          932.29
---------------Time to First Token----------------
Mean TTFT (ms):                          49.00
Median TTFT (ms):                        47.49
P99 TTFT (ms):                           107.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.31
Median TPOT (ms):                        8.24
P99 TPOT (ms):                           10.12
---------------Inter-token Latency----------------
Mean ITL (ms):                           8.40
Median ITL (ms):                         7.99
P99 ITL (ms):                            23.32
==================================================
优化后的推理服务脚本 
1
vllm serve openai/gpt-oss-120b --async-scheduling -tp 2
优化后的测试结果 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  98.86
Total input tokens:                      3200000
Total generated tokens:                  9955
Request throughput (req/s):              1.01
Output token throughput (tok/s):         100.70
Total Token throughput (tok/s):          32471.01
---------------Time to First Token----------------
Mean TTFT (ms):                          48826.05
Median TTFT (ms):                        48671.00
P99 TTFT (ms):                           96651.74
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          220.32
Median TPOT (ms):                        252.27
P99 TPOT (ms):                           252.79
---------------Inter-token Latency----------------
Mean ITL (ms):                           228.81
Median ITL (ms):                         246.94
P99 ITL (ms):                            481.43
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  59.94
Total input tokens:                      1999249
Total generated tokens:                  68046
Request throughput (req/s):              8.34
Output token throughput (tok/s):         1135.29
Total Token throughput (tok/s):          34491.13
---------------Time to First Token----------------
Mean TTFT (ms):                          29487.34
Median TTFT (ms):                        29235.61
P99 TTFT (ms):                           56201.04
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          164.73
Median TPOT (ms):                        199.03
P99 TPOT (ms):                           220.23
---------------Inter-token Latency----------------
Mean ITL (ms):                           171.11
Median ITL (ms):                         215.59
P99 ITL (ms):                            431.23
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  30.75
Total input tokens:                      999120
Total generated tokens:                  49059
Request throughput (req/s):              16.26
Output token throughput (tok/s):         1595.22
Total Token throughput (tok/s):          34083.02
---------------Time to First Token----------------
Mean TTFT (ms):                          15291.77
Median TTFT (ms):                        15170.31
P99 TTFT (ms):                           28437.70
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          139.32
Median TPOT (ms):                        148.04
P99 TPOT (ms):                           212.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           147.37
Median ITL (ms):                         208.40
P99 ITL (ms):                            421.37
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  4.18
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              239.21
Output token throughput (tok/s):         956.85
Total Token throughput (tok/s):          31517.47
---------------Time to First Token----------------
Mean TTFT (ms):                          2800.15
Median TTFT (ms):                        2412.28
P99 TTFT (ms):                           3856.95
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          183.05
Median TPOT (ms):                        208.66
P99 TPOT (ms):                           232.15
---------------Inter-token Latency----------------
Mean ITL (ms):                           274.85
Median ITL (ms):                         267.83
P99 ITL (ms):                            506.27
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  33.60
Total input tokens:                      22946
Total generated tokens:                  21691
Request throughput (req/s):              2.98
Output token throughput (tok/s):         645.55
Total Token throughput (tok/s):          1328.44
---------------Time to First Token----------------
Mean TTFT (ms):                          37.51
Median TTFT (ms):                        36.68
P99 TTFT (ms):                           58.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.75
Median TPOT (ms):                        5.74
P99 TPOT (ms):                           6.91
---------------Inter-token Latency----------------
Mean ITL (ms):                           5.89
Median ITL (ms):                         5.54
P99 ITL (ms):                            23.43
==================================================