Optimizing GPT-OSS-20B Throughput on NVIDIA A100 GPUs
Conclusion
Recommended configuration for optimizing throughput of GPT-OSS-20B on A100 GPUs:
Serving Command
vllm serve openai/gpt-oss-20b --async-scheduling
Comparison of benchmark results before and after optimization:
| Benchmark Case | baseline (vLLM without any optimizations) | Optimized |
|---|---|---|
| ShareGPT | Total TPS: 9743.28 Mean TPOT(ms): 48.57 |
Total TPS: 10919.92 (+12.1%) Mean TPOT(ms): 43.25 |
| Short Prompt | Total TPS: 24223.22 Mean TPOT(ms): 62.10 |
Total TPS: 25172.82 (+4.0%) Mean TPOT(ms): 55.31 |
| Medium Prompt | Total TPS: 18021.34 Mean TPOT(ms): 100.32 |
Total TPS: 18920.16 (+5.0%) Mean TPOT(ms): 95.80 |
| Long Prompt | Total TPS: 15938.87 Mean TPOT(ms): 123.35 |
Total TPS: 16550.81 (+3.8%) Mean TPOT(ms): 119.37 |
| Very Long Prompt | Total TPS: 7280.33 Mean TPOT(ms): 268.03 |
Total TPS: 7363.09 (+1.1%) Mean TPOT(ms): 264.93 |
Note
- Our benchmark tests do not cover all possible optimization combinations. For example, we select the inference engine that performs best under its default configuration as the starting point for further tuning. This pruning approach yields a local optimum, which may not be the global optimum.
- There are other optimization methods that depend on specific user scenarios, including max batch size, schedule configuration, extended KV cache, CUDA graph, Torch Compile, etc. The conclusions in this document can serve as a starting point for more targeted optimizations.
- The tests are conducted on specific hardware and software setups. Advances in the inference engine may lead to new conclusions.
If there are any missing points or updates reflecting new changes, please let us know.
Optimization Objective
Achieve high throughput under high-concurrency request scenarios.
Experimental Setup
Model
GPT-OSS-20B
Hardware
NVIDIA A100 GPUs
Engine Version
- vLLM: v0.11.0
- SGLang: v0.5.4.post2
- TensorRT-LLM: v1.2.0rc1
Benchmark Dataset
- ShareGPT
- Random dataset with varying sequence lengths:
- Very long prompt: 32000 input tokens, 100 output tokens
- Long prompt: 4000 input tokens, 200 output tokens
- Medium prompt: 2000 input tokens, 100 output tokens
- Short prompt: 128 input tokens, 4 output tokens
Benchmark Script
We use the vLLM bench CLI tool to benchmark the model performance. The following command is used to run the benchmark:
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
# Benchmark on ShareGPT dataset
vllm bench serve --model openai/gpt-oss-20b --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000
# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model openai/gpt-oss-20b --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42
Experiment Results
1. Choosing the Inference Engine
vLLM
Serving script
vllm serve openai/gpt-oss-20b
Benchmark result
============ Serving Benchmark Result ============
Successful requests: 1000
Benchmark duration (s): 42.03
Total input tokens: 215312
Total generated tokens: 194171
Request throughput (req/s): 23.79
Output token throughput (tok/s): 4620.12
Peak output token throughput (tok/s): 7484.00
Peak concurrent requests: 1000.00
Total Token throughput (tok/s): 9743.28
---------------Time to First Token----------------
Mean TTFT (ms): 13654.59
Median TTFT (ms): 13063.93
P99 TTFT (ms): 30306.72
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 48.57
Median TPOT (ms): 45.15
P99 TPOT (ms): 110.58
---------------Inter-token Latency----------------
Mean ITL (ms): 41.95
Median ITL (ms): 33.96
P99 ITL (ms): 112.27
==================================================
SGLang
Serving script
python3 -m sglang.launch_server --model-path openai/gpt-oss-20b
Benchmark result
============ Serving Benchmark Result ============
Successful requests: 1000
Benchmark duration (s): 735.43
Total input tokens: 215312
Total generated tokens: 195617
Request throughput (req/s): 1.36
Output token throughput (tok/s): 265.99
Peak output token throughput (tok/s): 5848.00
Peak concurrent requests: 1000.00
Total Token throughput (tok/s): 558.76
---------------Time to First Token----------------
Mean TTFT (ms): 140888.75
Median TTFT (ms): 137990.65
P99 TTFT (ms): 274130.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6243.98
Median TPOT (ms): 2508.45
P99 TPOT (ms): 46225.15
---------------Inter-token Latency----------------
Mean ITL (ms): 1889.78
Median ITL (ms): 1305.64
P99 ITL (ms): 4248.44
==================================================
TensorRT-LLM
Serving script
trtllm-serve openai/gpt-oss-120b --max_seq_len 32768
Benchmark result
NotImplementedError: WFP4A16 MoE is unsupported on SM80.
Result: vLLM (9743.28 tok/s) > SGLang(558.76 tok/s) > TensorRT-LLM (Not supported)
2. Async scheduling in vLLM
Serving script
vllm serve openai/gpt-oss-20b --async-scheduling
Benchmark result
============ Serving Benchmark Result ============
Successful requests: 1000
Benchmark duration (s): 37.51
Total input tokens: 215312
Total generated tokens: 194303
Request throughput (req/s): 26.66
Output token throughput (tok/s): 5179.92
Peak output token throughput (tok/s): 8545.00
Peak concurrent requests: 1000.00
Total Token throughput (tok/s): 10919.92
---------------Time to First Token----------------
Mean TTFT (ms): 12371.62
Median TTFT (ms): 11647.98
P99 TTFT (ms): 27040.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 43.25
Median TPOT (ms): 39.47
P99 TPOT (ms): 105.22
---------------Inter-token Latency----------------
Mean ITL (ms): 36.85
Median ITL (ms): 28.41
P99 ITL (ms): 106.78
==================================================
3. Tuning number of batched tokens in vLLM
Serving script
vllm serve openai/gpt-oss-20b --async-scheduling --max-num-batched-tokens 8192
Benchmark result
# --max-num-batched-tokens 4096
============ Serving Benchmark Result ============
Successful requests: 1000
Benchmark duration (s): 37.43
Total input tokens: 215312
Total generated tokens: 194145
Request throughput (req/s): 26.72
Output token throughput (tok/s): 5186.81
Peak output token throughput (tok/s): 8601.00
Peak concurrent requests: 1000.00
Total Token throughput (tok/s): 10939.12
---------------Time to First Token----------------
Mean TTFT (ms): 12298.27
Median TTFT (ms): 11609.21
P99 TTFT (ms): 27015.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 44.99
Median TPOT (ms): 39.55
P99 TPOT (ms): 195.62
---------------Inter-token Latency----------------
Mean ITL (ms): 37.00
Median ITL (ms): 28.57
P99 ITL (ms): 118.90
==================================================
# --max-num-batched-tokens 8192
============ Serving Benchmark Result ============
Successful requests: 1000
Benchmark duration (s): 37.58
Total input tokens: 215312
Total generated tokens: 193986
Request throughput (req/s): 26.61
Output token throughput (tok/s): 5161.85
Peak output token throughput (tok/s): 8663.00
Peak concurrent requests: 1000.00
Total Token throughput (tok/s): 10891.17
---------------Time to First Token----------------
Mean TTFT (ms): 12479.83
Median TTFT (ms): 11828.63
P99 TTFT (ms): 27160.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 45.94
Median TPOT (ms): 39.39
P99 TPOT (ms): 234.47
---------------Inter-token Latency----------------
Mean ITL (ms): 36.95
Median ITL (ms): 28.33
P99 ITL (ms): 101.21
==================================================
Summary of Optimization Options
| Optimization Option | Throughput Improvement |
|---|---|
| Engine Selection | - |
| Async Scheduling | +12.1% |
| Batched Tokens | - |
Other Benchmark Cases
We further benchmarked the optimized configuration to evaluate its generalization under various workloads.
Baseline serving script
vllm serve openai/gpt-oss-20b
Baseline benchmark results
# random 32K input
============ Serving Benchmark Result ============
Successful requests: 100
Benchmark duration (s): 440.85
Total input tokens: 3200000
Total generated tokens: 9533
Request throughput (req/s): 0.23
Output token throughput (tok/s): 21.62
Peak output token throughput (tok/s): 203.00
Peak concurrent requests: 100.00
Total Token throughput (tok/s): 7280.33
---------------Time to First Token----------------
Mean TTFT (ms): 219739.77
Median TTFT (ms): 219663.48
P99 TTFT (ms): 435080.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 268.03
Median TPOT (ms): 278.17
P99 TPOT (ms): 282.40
---------------Inter-token Latency----------------
Mean ITL (ms): 276.43
Median ITL (ms): 274.32
P99 ITL (ms): 438.91
==================================================
# random 4K input
============ Serving Benchmark Result ============
Successful requests: 500
Benchmark duration (s): 128.79
Total input tokens: 1999249
Total generated tokens: 53530
Request throughput (req/s): 3.88
Output token throughput (tok/s): 415.64
Peak output token throughput (tok/s): 1983.00
Peak concurrent requests: 500.00
Total Token throughput (tok/s): 15938.87
---------------Time to First Token----------------
Mean TTFT (ms): 62947.13
Median TTFT (ms): 61941.78
P99 TTFT (ms): 125429.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 123.35
Median TPOT (ms): 129.79
P99 TPOT (ms): 131.42
---------------Inter-token Latency----------------
Mean ITL (ms): 130.61
Median ITL (ms): 129.50
P99 ITL (ms): 262.38
==================================================
# random 2K input
============ Serving Benchmark Result ============
Successful requests: 500
Benchmark duration (s): 58.06
Total input tokens: 999120
Total generated tokens: 47224
Request throughput (req/s): 8.61
Output token throughput (tok/s): 813.35
Peak output token throughput (tok/s): 4701.00
Peak concurrent requests: 500.00
Total Token throughput (tok/s): 18021.34
---------------Time to First Token----------------
Mean TTFT (ms): 31669.55
Median TTFT (ms): 32083.12
P99 TTFT (ms): 55891.47
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 100.32
Median TPOT (ms): 119.20
P99 TPOT (ms): 119.71
---------------Inter-token Latency----------------
Mean ITL (ms): 106.87
Median ITL (ms): 119.09
P99 ITL (ms): 239.78
==================================================
# random 128 input
============ Serving Benchmark Result ============
Successful requests: 1000
Benchmark duration (s): 5.44
Total input tokens: 127755
Total generated tokens: 4000
Request throughput (req/s): 183.85
Output token throughput (tok/s): 735.40
Peak output token throughput (tok/s): 932.00
Peak concurrent requests: 1000.00
Total Token throughput (tok/s): 24223.22
---------------Time to First Token----------------
Mean TTFT (ms): 2774.46
Median TTFT (ms): 1971.28
P99 TTFT (ms): 5298.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 62.10
Median TPOT (ms): 65.44
P99 TPOT (ms): 106.62
---------------Inter-token Latency----------------
Mean ITL (ms): 100.20
Median ITL (ms): 105.43
P99 ITL (ms): 213.73
==================================================
Optimized serving script
vllm serve openai/gpt-oss-20b --async-scheduling
Optimized benchmark results
# random 32K input
============ Serving Benchmark Result ============
Successful requests: 100
Benchmark duration (s): 435.88
Total input tokens: 3200000
Total generated tokens: 9394
Request throughput (req/s): 0.23
Output token throughput (tok/s): 21.55
Peak output token throughput (tok/s): 239.00
Peak concurrent requests: 100.00
Total Token throughput (tok/s): 7363.09
---------------Time to First Token----------------
Mean TTFT (ms): 217408.40
Median TTFT (ms): 217388.75
P99 TTFT (ms): 430331.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 264.93
Median TPOT (ms): 275.04
P99 TPOT (ms): 278.94
---------------Inter-token Latency----------------
Mean ITL (ms): 274.36
Median ITL (ms): 274.48
P99 ITL (ms): 433.63
==================================================
# random 4K input
============ Serving Benchmark Result ============
Successful requests: 500
Benchmark duration (s): 123.99
Total input tokens: 1999249
Total generated tokens: 52857
Request throughput (req/s): 4.03
Output token throughput (tok/s): 426.31
Peak output token throughput (tok/s): 1699.00
Peak concurrent requests: 500.00
Total Token throughput (tok/s): 16550.81
---------------Time to First Token----------------
Mean TTFT (ms): 60587.64
Median TTFT (ms): 59590.47
P99 TTFT (ms): 121079.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 119.37
Median TPOT (ms): 125.63
P99 TPOT (ms): 126.57
---------------Inter-token Latency----------------
Mean ITL (ms): 126.46
Median ITL (ms): 125.29
P99 ITL (ms): 253.57
==================================================
# random 2K input
============ Serving Benchmark Result ============
Successful requests: 500
Benchmark duration (s): 55.29
Total input tokens: 999120
Total generated tokens: 46961
Request throughput (req/s): 9.04
Output token throughput (tok/s): 849.37
Peak output token throughput (tok/s): 4961.00
Peak concurrent requests: 500.00
Total Token throughput (tok/s): 18920.16
---------------Time to First Token----------------
Mean TTFT (ms): 30030.76
Median TTFT (ms): 30306.12
P99 TTFT (ms): 53186.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 95.80
Median TPOT (ms): 114.20
P99 TPOT (ms): 114.34
---------------Inter-token Latency----------------
Mean ITL (ms): 102.24
Median ITL (ms): 114.01
P99 ITL (ms): 229.30
==================================================
# random 128 input
============ Serving Benchmark Result ============
Successful requests: 1000
Benchmark duration (s): 5.23
Total input tokens: 127755
Total generated tokens: 4000
Request throughput (req/s): 191.06
Output token throughput (tok/s): 764.23
Peak output token throughput (tok/s): 1302.00
Peak concurrent requests: 1000.00
Total Token throughput (tok/s): 25172.82
---------------Time to First Token----------------
Mean TTFT (ms): 2682.01
Median TTFT (ms): 1944.72
P99 TTFT (ms): 5078.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 55.31
Median TPOT (ms): 46.59
P99 TPOT (ms): 102.22
---------------Inter-token Latency----------------
Mean ITL (ms): 85.84
Median ITL (ms): 100.82
P99 ITL (ms): 205.12
==================================================