跳转至

NVIDIA A100 GPUs 上的 GPT-OSS-20B 吞吐量优化

结论

gpt-oss-20b-a100

GPT-OSS-20B 在 A100 GPUs 上的吞吐量优化推荐配置：

推理服务命令 
1
vllm serve openai/gpt-oss-20b --async-scheduling

优化前后的基准测试结果对比：

基准测试用例 基准配置（未做任何优化的 vLLM） 优化后
ShareGPT Total TPS: 9743.28
Mean TPOT(ms): 48.57		 Total TPS: 10919.92 (+12.1%)
Mean TPOT(ms): 43.25
短提示词 Total TPS: 24223.22
Mean TPOT(ms): 62.10		 Total TPS: 25172.82 (+4.0%)
Mean TPOT(ms): 55.31
中等长度提示词 Total TPS: 18021.34
Mean TPOT(ms): 100.32		 Total TPS: 18920.16 (+5.0%)
Mean TPOT(ms): 95.80
长提示词 Total TPS: 15938.87
Mean TPOT(ms): 123.35		 Total TPS: 16550.81 (+3.8%)
Mean TPOT(ms): 119.37
超长提示词 Total TPS: 7280.33
Mean TPOT(ms): 268.03		 Total TPS: 7363.09 (+1.1%)
Mean TPOT(ms): 264.93

说明

  1. 基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
  2. 其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph、Torch Compile 等。本文结论可作为进一步针对性优化的起点。
  3. 测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

优化目标

在高并发请求场景下实现高吞吐量。

实验环境

模型

GPT-OSS-20B

硬件

单个 NVIDIA A100 SXM 80GB GPU。

引擎版本

  • vLLM: v0.11.0
  • SGLang: v0.5.4.post2
  • TensorRT-LLM: v1.2.0rc1

基准测试数据集

  1. ShareGPT
  2. 不同序列长度的随机数据集：
    • 超长提示词：32000 输入 tokens，100 输出 tokens
    • 长提示词：4000 输入 tokens，200 输出 tokens
    • 中等长度提示词：2000 输入 tokens，100 输出 tokens
    • 短提示词：128 输入 tokens，4 输出 tokens

基准测试脚本

使用 vLLM bench CLI 工具对模型性能进行基准测试。运行基准测试的命令如下：

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model openai/gpt-oss-20b --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model openai/gpt-oss-20b --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

实验结果

1. 选择推理引擎

vLLM

推理服务脚本 
1
vllm serve openai/gpt-oss-20b
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  42.03
Total input tokens:                      215312
Total generated tokens:                  194171
Request throughput (req/s):              23.79
Output token throughput (tok/s):         4620.12
Peak output token throughput (tok/s):    7484.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          9743.28
---------------Time to First Token----------------
Mean TTFT (ms):                          13654.59
Median TTFT (ms):                        13063.93
P99 TTFT (ms):                           30306.72
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          48.57
Median TPOT (ms):                        45.15
P99 TPOT (ms):                           110.58
---------------Inter-token Latency----------------
Mean ITL (ms):                           41.95
Median ITL (ms):                         33.96
P99 ITL (ms):                            112.27
==================================================

SGLang

推理服务脚本 
1
python3 -m sglang.launch_server --model-path openai/gpt-oss-20b
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  735.43
Total input tokens:                      215312
Total generated tokens:                  195617
Request throughput (req/s):              1.36
Output token throughput (tok/s):         265.99
Peak output token throughput (tok/s):    5848.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          558.76
---------------Time to First Token----------------
Mean TTFT (ms):                          140888.75
Median TTFT (ms):                        137990.65
P99 TTFT (ms):                           274130.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6243.98
Median TPOT (ms):                        2508.45
P99 TPOT (ms):                           46225.15
---------------Inter-token Latency----------------
Mean ITL (ms):                           1889.78
Median ITL (ms):                         1305.64
P99 ITL (ms):                            4248.44
==================================================

TensorRT-LLM

推理服务脚本 
1
trtllm-serve openai/gpt-oss-120b --max_seq_len 32768
基准测试结果 
1
NotImplementedError: WFP4A16 MoE is unsupported on SM80.

结果：vLLM (9743.28 tok/s) > SGLang(558.76 tok/s) > TensorRT-LLM (不支持)

2. vLLM 中的 异步调度

推理服务脚本 
1
vllm serve openai/gpt-oss-20b --async-scheduling
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  37.51
Total input tokens:                      215312
Total generated tokens:                  194303
Request throughput (req/s):              26.66
Output token throughput (tok/s):         5179.92
Peak output token throughput (tok/s):    8545.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10919.92
---------------Time to First Token----------------
Mean TTFT (ms):                          12371.62
Median TTFT (ms):                        11647.98
P99 TTFT (ms):                           27040.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          43.25
Median TPOT (ms):                        39.47
P99 TPOT (ms):                           105.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           36.85
Median ITL (ms):                         28.41
P99 ITL (ms):                            106.78
==================================================

3. vLLM 中的 批处理 Token 数调优

推理服务脚本 
1
vllm serve openai/gpt-oss-20b --async-scheduling --max-num-batched-tokens 8192
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# --max-num-batched-tokens 4096
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  37.43
Total input tokens:                      215312
Total generated tokens:                  194145
Request throughput (req/s):              26.72
Output token throughput (tok/s):         5186.81
Peak output token throughput (tok/s):    8601.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10939.12
---------------Time to First Token----------------
Mean TTFT (ms):                          12298.27
Median TTFT (ms):                        11609.21
P99 TTFT (ms):                           27015.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          44.99
Median TPOT (ms):                        39.55
P99 TPOT (ms):                           195.62
---------------Inter-token Latency----------------
Mean ITL (ms):                           37.00
Median ITL (ms):                         28.57
P99 ITL (ms):                            118.90
==================================================
# --max-num-batched-tokens 8192
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  37.58
Total input tokens:                      215312
Total generated tokens:                  193986
Request throughput (req/s):              26.61
Output token throughput (tok/s):         5161.85
Peak output token throughput (tok/s):    8663.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10891.17
---------------Time to First Token----------------
Mean TTFT (ms):                          12479.83
Median TTFT (ms):                        11828.63
P99 TTFT (ms):                           27160.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          45.94
Median TPOT (ms):                        39.39
P99 TPOT (ms):                           234.47
---------------Inter-token Latency----------------
Mean ITL (ms):                           36.95
Median ITL (ms):                         28.33
P99 ITL (ms):                            101.21
==================================================

优化选项汇总

优化选项 吞吐量提升
引擎选择 -
异步调度 +12.1%
批处理 Token 数 -

其他基准测试用例

我们进一步对优化后的配置进行了基准测试，评估其在不同工作负载下的泛化表现。

基准配置的推理服务脚本 
1
vllm serve openai/gpt-oss-20b
基准配置的测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  440.85
Total input tokens:                      3200000
Total generated tokens:                  9533
Request throughput (req/s):              0.23
Output token throughput (tok/s):         21.62
Peak output token throughput (tok/s):    203.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          7280.33
---------------Time to First Token----------------
Mean TTFT (ms):                          219739.77
Median TTFT (ms):                        219663.48
P99 TTFT (ms):                           435080.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          268.03
Median TPOT (ms):                        278.17
P99 TPOT (ms):                           282.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           276.43
Median ITL (ms):                         274.32
P99 ITL (ms):                            438.91
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  128.79
Total input tokens:                      1999249
Total generated tokens:                  53530
Request throughput (req/s):              3.88
Output token throughput (tok/s):         415.64
Peak output token throughput (tok/s):    1983.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          15938.87
---------------Time to First Token----------------
Mean TTFT (ms):                          62947.13
Median TTFT (ms):                        61941.78
P99 TTFT (ms):                           125429.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          123.35
Median TPOT (ms):                        129.79
P99 TPOT (ms):                           131.42
---------------Inter-token Latency----------------
Mean ITL (ms):                           130.61
Median ITL (ms):                         129.50
P99 ITL (ms):                            262.38
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  58.06
Total input tokens:                      999120
Total generated tokens:                  47224
Request throughput (req/s):              8.61
Output token throughput (tok/s):         813.35
Peak output token throughput (tok/s):    4701.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          18021.34
---------------Time to First Token----------------
Mean TTFT (ms):                          31669.55
Median TTFT (ms):                        32083.12
P99 TTFT (ms):                           55891.47
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          100.32
Median TPOT (ms):                        119.20
P99 TPOT (ms):                           119.71
---------------Inter-token Latency----------------
Mean ITL (ms):                           106.87
Median ITL (ms):                         119.09
P99 ITL (ms):                            239.78
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  5.44
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              183.85
Output token throughput (tok/s):         735.40
Peak output token throughput (tok/s):    932.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          24223.22
---------------Time to First Token----------------
Mean TTFT (ms):                          2774.46
Median TTFT (ms):                        1971.28
P99 TTFT (ms):                           5298.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          62.10
Median TPOT (ms):                        65.44
P99 TPOT (ms):                           106.62
---------------Inter-token Latency----------------
Mean ITL (ms):                           100.20
Median ITL (ms):                         105.43
P99 ITL (ms):                            213.73
==================================================
优化后的推理服务脚本 
1
vllm serve openai/gpt-oss-20b --async-scheduling
优化后的测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  435.88
Total input tokens:                      3200000
Total generated tokens:                  9394
Request throughput (req/s):              0.23
Output token throughput (tok/s):         21.55
Peak output token throughput (tok/s):    239.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          7363.09
---------------Time to First Token----------------
Mean TTFT (ms):                          217408.40
Median TTFT (ms):                        217388.75
P99 TTFT (ms):                           430331.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          264.93
Median TPOT (ms):                        275.04
P99 TPOT (ms):                           278.94
---------------Inter-token Latency----------------
Mean ITL (ms):                           274.36
Median ITL (ms):                         274.48
P99 ITL (ms):                            433.63
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  123.99
Total input tokens:                      1999249
Total generated tokens:                  52857
Request throughput (req/s):              4.03
Output token throughput (tok/s):         426.31
Peak output token throughput (tok/s):    1699.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          16550.81
---------------Time to First Token----------------
Mean TTFT (ms):                          60587.64
Median TTFT (ms):                        59590.47
P99 TTFT (ms):                           121079.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          119.37
Median TPOT (ms):                        125.63
P99 TPOT (ms):                           126.57
---------------Inter-token Latency----------------
Mean ITL (ms):                           126.46
Median ITL (ms):                         125.29
P99 ITL (ms):                            253.57
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  55.29
Total input tokens:                      999120
Total generated tokens:                  46961
Request throughput (req/s):              9.04
Output token throughput (tok/s):         849.37
Peak output token throughput (tok/s):    4961.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          18920.16
---------------Time to First Token----------------
Mean TTFT (ms):                          30030.76
Median TTFT (ms):                        30306.12
P99 TTFT (ms):                           53186.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          95.80
Median TPOT (ms):                        114.20
P99 TPOT (ms):                           114.34
---------------Inter-token Latency----------------
Mean ITL (ms):                           102.24
Median ITL (ms):                         114.01
P99 ITL (ms):                            229.30
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  5.23
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              191.06
Output token throughput (tok/s):         764.23
Peak output token throughput (tok/s):    1302.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          25172.82
---------------Time to First Token----------------
Mean TTFT (ms):                          2682.01
Median TTFT (ms):                        1944.72
P99 TTFT (ms):                           5078.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          55.31
Median TPOT (ms):                        46.59
P99 TPOT (ms):                           102.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           85.84
Median ITL (ms):                         100.82
P99 ITL (ms):                            205.12
==================================================