NVIDIA H100 GPUs 上的 Qwen3-8B 延迟优化

结论

Qwen3-8B 在 H100 GPUs 上的延迟优化推荐配置：

推理服务命令

python3 -m sglang.launch_server \
    --model Qwen/Qwen3-8B-FP8 \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 \
    --speculative-num-steps 4 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16

优化前后的基准测试结果对比：

基准测试用例	基准配置（未做任何优化的 vLLM）	优化后
ShareGPT BS=1	Mean latency: 1.66s/req	Mean latency: 0.62s/req (2.7x，更快)
ShareGPT BS=2	Mean latency: 1.68s/req	Mean latency: 0.71s/req (2.4x，更快)
ShareGPT BS=4	Mean latency: 1.68s/req	Mean latency: 0.77s/req (2.2x，更快)
ShareGPT BS=8	Mean latency: 1.78s/req	Mean latency: 1.10s/req (1.6x，更快)

说明

基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph 等。本文结论可作为进一步针对性优化的起点。
测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。
量化可能影响准确率，但对于大多数模型，FP8 量化的准确率下降可控制在 1% 以内。详情参见评估结果。因此，强烈建议在低延迟推理服务场景中使用 FP8 量化。
推测解码能够显著降低低并发请求的延迟，但加速效果取决于基准测试数据集的数据分布和草稿模型的选择。例如，本测试所选的草稿模型使用英文数据训练，在其他语言上的表现可能不够理想。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

优化目标

在低并发（批大小 < 8）的请求场景下实现低延迟。

实验环境

模型

Qwen/Qwen3-8B

硬件

单个 NVIDIA H100 SXM GPU。

引擎版本

vLLM: v0.11.0
SGLang: v0.5.3.post3
TensorRT-LLM: v1.2.0rc1

基准测试数据集

ShareGPT

基准测试脚本

使用 vLLM bench CLI 工具对模型性能进行基准测试。运行基准测试的命令如下：

# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-8B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 100 --max-concurrency 4

实验结果

1. 选择推理引擎

vLLM

推理服务脚本

1	`vllm serve Qwen/Qwen3-8B`

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  44.40
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              2.25
Output token throughput (tok/s):         496.88
Peak output token throughput (tok/s):    540.00
Peak concurrent requests:                13.00
Total Token throughput (tok/s):          1020.76
---------------Time to First Token----------------
Mean TTFT (ms):                          64.00
Median TTFT (ms):                        60.26
P99 TTFT (ms):                           98.59
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.38
Median TPOT (ms):                        7.40
P99 TPOT (ms):                           7.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.78
Median ITL (ms):                         7.37
P99 ITL (ms):                            40.98
==================================================

SGLang

推理服务脚本

python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  41.97
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              2.38
Output token throughput (tok/s):         525.59
Peak output token throughput (tok/s):    572.00
Peak concurrent requests:                12.00
Total Token throughput (tok/s):          1079.74
---------------Time to First Token----------------
Mean TTFT (ms):                          27.70
Median TTFT (ms):                        27.25
P99 TTFT (ms):                           43.35
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.19
Median TPOT (ms):                        7.14
P99 TPOT (ms):                           7.90
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.08
Median ITL (ms):                         7.04
P99 ITL (ms):                            11.48
==================================================

TensorRT-LLM

推理服务脚本

1	`trtllm-serve Qwen/Qwen3-8B`

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  45.26
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              2.21
Output token throughput (tok/s):         487.38
Peak output token throughput (tok/s):    541.00
Peak concurrent requests:                11.00
Total Token throughput (tok/s):          1001.25
---------------Time to First Token----------------
Mean TTFT (ms):                          51.61
Median TTFT (ms):                        45.46
P99 TTFT (ms):                           134.71
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.67
Median TPOT (ms):                        7.58
P99 TPOT (ms):                           10.15
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.59
Median ITL (ms):                         7.41
P99 ITL (ms):                            24.38
==================================================

每个请求的平均延迟：SGLang(1.61 s) < vLLM (1.68 s) > TensorRT-LLM (1.74 s)

2. SGLang 中的 CUDA graph 最大批大小

推理服务脚本

python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B --cuda-graph-max-bs 8

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  42.04
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              2.38
Output token throughput (tok/s):         524.71
Peak output token throughput (tok/s):    572.00
Peak concurrent requests:                11.00
Total Token throughput (tok/s):          1077.94
---------------Time to First Token----------------
Mean TTFT (ms):                          28.46
Median TTFT (ms):                        27.35
P99 TTFT (ms):                           39.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.20
Median TPOT (ms):                        7.15
P99 TPOT (ms):                           8.13
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.09
Median ITL (ms):                         7.04
P99 ITL (ms):                            11.56
==================================================

3. 量化

推理服务脚本

python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B-FP8

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  33.54
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              2.98
Output token throughput (tok/s):         657.83
Peak output token throughput (tok/s):    741.00
Peak concurrent requests:                11.00
Total Token throughput (tok/s):          1351.41
---------------Time to First Token----------------
Mean TTFT (ms):                          39.66
Median TTFT (ms):                        33.38
P99 TTFT (ms):                           134.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.65
Median TPOT (ms):                        5.58
P99 TPOT (ms):                           7.35
---------------Inter-token Latency----------------
Mean ITL (ms):                           5.58
Median ITL (ms):                         5.39
P99 ITL (ms):                            22.20
==================================================

4. 注意力后端

推理服务脚本

python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B-FP8 --attention-backend triton

基准测试结果

# triton
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  34.73
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              2.88
Output token throughput (tok/s):         635.15
Peak output token throughput (tok/s):    744.00
Peak concurrent requests:                10.00
Total Token throughput (tok/s):          1304.82
---------------Time to First Token----------------
Mean TTFT (ms):                          60.18
Median TTFT (ms):                        43.84
P99 TTFT (ms):                           493.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.77
Median TPOT (ms):                        5.63
P99 TPOT (ms):                           7.78
---------------Inter-token Latency----------------
Mean ITL (ms):                           5.71
Median ITL (ms):                         5.40
P99 ITL (ms):                            28.58
==================================================

# flashinfer
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  39.63
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              2.52
Output token throughput (tok/s):         556.61
Peak output token throughput (tok/s):    728.00
Peak concurrent requests:                11.00
Total Token throughput (tok/s):          1143.47
---------------Time to First Token----------------
Mean TTFT (ms):                          269.33
Median TTFT (ms):                        40.70
P99 TTFT (ms):                           5700.99
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.71
Median TPOT (ms):                        5.63
P99 TPOT (ms):                           7.44
---------------Inter-token Latency----------------
Mean ITL (ms):                           5.65
Median ITL (ms):                         5.40
P99 ITL (ms):                            28.71
==================================================

5. 推测解码

推理服务脚本

python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B-FP8 \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 \
    --speculative-num-steps 4 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16

基准测试结果

# --speculative-algorithm EAGLE3 --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 --speculative-num-steps 4 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  22.53
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              4.44
Output token throughput (tok/s):         979.24
Peak output token throughput (tok/s):    393.00
Peak concurrent requests:                15.00
Total Token throughput (tok/s):          2011.70
---------------Time to First Token----------------
Mean TTFT (ms):                          50.70
Median TTFT (ms):                        43.71
P99 TTFT (ms):                           368.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.75
Median TPOT (ms):                        3.32
P99 TPOT (ms):                           9.63
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.05
Median ITL (ms):                         9.86
P99 ITL (ms):                            38.76
==================================================

# --speculative-algorithm EAGLE3 --speculative-draft-model-path AngelSlim/Qwen3-8B_eagle3 --speculative-num-steps 4 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  26.89
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              3.72
Output token throughput (tok/s):         820.28
Peak output token throughput (tok/s):    404.00
Peak concurrent requests:                13.00
Total Token throughput (tok/s):          1685.15
---------------Time to First Token----------------
Mean TTFT (ms):                          44.60
Median TTFT (ms):                        44.68
P99 TTFT (ms):                           78.96
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.45
Median TPOT (ms):                        4.40
P99 TPOT (ms):                           7.91
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.49
Median ITL (ms):                         9.88
P99 ITL (ms):                            38.17
==================================================

# --speculative-algorithm EAGLE3 --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 --speculative-num-steps 2 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  24.16
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              4.14
Output token throughput (tok/s):         913.20
Peak output token throughput (tok/s):    433.00
Peak concurrent requests:                12.00
Total Token throughput (tok/s):          1876.03
---------------Time to First Token----------------
Mean TTFT (ms):                          49.89
Median TTFT (ms):                        44.38
P99 TTFT (ms):                           277.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.98
Median TPOT (ms):                        3.74
P99 TPOT (ms):                           7.81
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.82
Median ITL (ms):                         8.86
P99 ITL (ms):                            38.40
==================================================
# --speculative-algorithm EAGLE3 --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 --speculative-num-steps 8 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  27.17
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              3.68
Output token throughput (tok/s):         811.85
Peak output token throughput (tok/s):    336.00
Peak concurrent requests:                11.00
Total Token throughput (tok/s):          1667.83
---------------Time to First Token----------------
Mean TTFT (ms):                          64.58
Median TTFT (ms):                        56.79
P99 TTFT (ms):                           366.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.27
Median TPOT (ms):                        4.00
P99 TPOT (ms):                           8.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           13.94
Median ITL (ms):                         12.20
P99 ITL (ms):                            63.04
==================================================
# --speculative-algorithm EAGLE3 --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 --speculative-num-steps 4 --speculative-eagle-topk 8 --speculative-num-draft-tokens 16
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  22.72
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              4.40
Output token throughput (tok/s):         971.03
Peak output token throughput (tok/s):    382.00
Peak concurrent requests:                14.00
Total Token throughput (tok/s):          1994.83
---------------Time to First Token----------------
Mean TTFT (ms):                          49.08
Median TTFT (ms):                        36.31
P99 TTFT (ms):                           335.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.62
Median TPOT (ms):                        3.36
P99 TPOT (ms):                           9.61
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.24
Median ITL (ms):                         10.14
P99 ITL (ms):                            41.05
==================================================

# --speculative-algorithm EAGLE3 --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 --speculative-num-steps 4 --speculative-eagle-topk 4 --speculative-num-draft-tokens 32
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  22.68
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              4.41
Output token throughput (tok/s):         972.69
Peak output token throughput (tok/s):    354.00
Peak concurrent requests:                15.00
Total Token throughput (tok/s):          1998.25
---------------Time to First Token----------------
Mean TTFT (ms):                          50.97
Median TTFT (ms):                        40.01
P99 TTFT (ms):                           361.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.54
Median TPOT (ms):                        3.38
P99 TPOT (ms):                           7.82
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.90
Median ITL (ms):                         10.67
P99 ITL (ms):                            40.09
==================================================


# --speculative-algorithm EAGLE3 --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 --speculative-num-steps 4 --speculative-eagle-topk 4 --speculative-num-draft-tokens 8
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  23.94
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              4.18
Output token throughput (tok/s):         921.47
Peak output token throughput (tok/s):    399.00
Peak concurrent requests:                14.00
Total Token throughput (tok/s):          1893.03
---------------Time to First Token----------------
Mean TTFT (ms):                          47.83
Median TTFT (ms):                        45.01
P99 TTFT (ms):                           255.59
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.71
Median TPOT (ms):                        3.59
P99 TPOT (ms):                           6.20
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.71
Median ITL (ms):                         9.78
P99 ITL (ms):                            39.48
==================================================

# --speculative-algorithm EAGLE3 --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  26.70
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              3.75
Output token throughput (tok/s):         826.28
Peak output token throughput (tok/s):    521.00
Peak concurrent requests:                12.00
Total Token throughput (tok/s):          1697.47
---------------Time to First Token----------------
Mean TTFT (ms):                          56.09
Median TTFT (ms):                        44.66
P99 TTFT (ms):                           327.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.37
Median TPOT (ms):                        4.10
P99 TPOT (ms):                           7.78
---------------Inter-token Latency----------------
Mean ITL (ms):                           8.78
Median ITL (ms):                         7.62
P99 ITL (ms):                            48.17
==================================================

# --speculative-algorithm EAGLE3 --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  25.80
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              3.88
Output token throughput (tok/s):         854.98
Peak output token throughput (tok/s):    392.00
Peak concurrent requests:                11.00
Total Token throughput (tok/s):          1756.43
---------------Time to First Token----------------
Mean TTFT (ms):                          53.61
Median TTFT (ms):                        45.08
P99 TTFT (ms):                           374.04
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.07
Median TPOT (ms):                        3.68
P99 TPOT (ms):                           12.12
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.69
Median ITL (ms):                         10.35
P99 ITL (ms):                            40.08
==================================================

6. Torch.compile

推理服务脚本

python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B-FP8 \
  --enable-torch-compile \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 \
  --speculative-num-steps 4 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16

基准测试结果

============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  23.14
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              4.32
Output token throughput (tok/s):         953.40
Peak output token throughput (tok/s):    386.00
Peak concurrent requests:                14.00
Total Token throughput (tok/s):          1958.61
---------------Time to First Token----------------
Mean TTFT (ms):                          54.86
Median TTFT (ms):                        45.34
P99 TTFT (ms):                           410.48
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.53
Median TPOT (ms):                        3.30
P99 TPOT (ms):                           9.67
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.38
Median ITL (ms):                         9.96
P99 ITL (ms):                            41.21
==================================================

优化选项汇总

优化选项	吞吐量提升
引擎选择	+4.3%
量化	+25.8%
推测解码	+66.2%
CUDA graph 最大批大小	-
注意力后端	-
Torch.compile	-

其他基准测试用例

我们进一步使用不同批大小对优化后的配置进行了基准测试。

基准配置的推理服务脚本

1	`vllm serve Qwen/Qwen3-8B`

基准配置的测试结果

# BS=1
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             1
Benchmark duration (s):                  166.04
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              0.60
Output token throughput (tok/s):         132.86
Peak output token throughput (tok/s):    140.00
Peak concurrent requests:                5.00
Total Token throughput (tok/s):          272.95
---------------Time to First Token----------------
Mean TTFT (ms):                          59.01
Median TTFT (ms):                        60.38
P99 TTFT (ms):                           73.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.28
Median TPOT (ms):                        7.22
P99 TPOT (ms):                           7.52
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.26
Median ITL (ms):                         7.23
P99 ITL (ms):                            8.17
==================================================

# BS=2
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             2
Benchmark duration (s):                  85.78
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              1.17
Output token throughput (tok/s):         257.19
Peak output token throughput (tok/s):    277.00
Peak concurrent requests:                7.00
Total Token throughput (tok/s):          528.35
---------------Time to First Token----------------
Mean TTFT (ms):                          63.78
Median TTFT (ms):                        64.78
P99 TTFT (ms):                           93.70
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.37
Median TPOT (ms):                        7.30
P99 TPOT (ms):                           8.57
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.44
Median ITL (ms):                         7.28
P99 ITL (ms):                            8.75
==================================================

# BS=4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  44.40
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              2.25
Output token throughput (tok/s):         496.88
Peak output token throughput (tok/s):    540.00
Peak concurrent requests:                13.00
Total Token throughput (tok/s):          1020.76
---------------Time to First Token----------------
Mean TTFT (ms):                          64.00
Median TTFT (ms):                        60.26
P99 TTFT (ms):                           98.59
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.38
Median TPOT (ms):                        7.40
P99 TPOT (ms):                           7.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           7.78
Median ITL (ms):                         7.37
P99 ITL (ms):                            40.98
==================================================

# BS=8
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             8
Benchmark duration (s):                  24.05
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              4.16
Output token throughput (tok/s):         917.28
Peak output token throughput (tok/s):    1039.00
Peak concurrent requests:                15.00
Total Token throughput (tok/s):          1884.42
---------------Time to First Token----------------
Mean TTFT (ms):                          74.57
Median TTFT (ms):                        60.77
P99 TTFT (ms):                           204.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.71
Median TPOT (ms):                        7.67
P99 TPOT (ms):                           8.19
---------------Inter-token Latency----------------
Mean ITL (ms):                           8.57
Median ITL (ms):                         7.63
P99 ITL (ms):                            45.48
==================================================

优化后的推理服务脚本

python3 -m sglang.launch_server \
  --model Qwen/Qwen3-8B-FP8 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path Tengyunw/qwen3_8b_eagle3 \
  --speculative-num-steps 4 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16

优化后的测试结果

# BS=1
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             1
Benchmark duration (s):                  65.82
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              1.52
Output token throughput (tok/s):         335.15
Peak output token throughput (tok/s):    112.00
Peak concurrent requests:                6.00
Total Token throughput (tok/s):          688.51
---------------Time to First Token----------------
Mean TTFT (ms):                          34.70
Median TTFT (ms):                        34.11
P99 TTFT (ms):                           40.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.67
Median TPOT (ms):                        2.63
P99 TPOT (ms):                           3.46
---------------Inter-token Latency----------------
Mean ITL (ms):                           8.74
Median ITL (ms):                         8.95
P99 ITL (ms):                            10.46
==================================================

# BS=2
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             2
Benchmark duration (s):                  37.62
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              2.66
Output token throughput (tok/s):         586.35
Peak output token throughput (tok/s):    210.00
Peak concurrent requests:                8.00
Total Token throughput (tok/s):          1204.57
---------------Time to First Token----------------
Mean TTFT (ms):                          39.41
Median TTFT (ms):                        41.19
P99 TTFT (ms):                           57.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.04
Median TPOT (ms):                        2.96
P99 TPOT (ms):                           5.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.69
Median ITL (ms):                         9.57
P99 ITL (ms):                            34.00
==================================================

# BS=4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  21.09
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              4.74
Output token throughput (tok/s):         1046.22
Peak output token throughput (tok/s):    387.00
Peak concurrent requests:                14.00
Total Token throughput (tok/s):          2149.30
---------------Time to First Token----------------
Mean TTFT (ms):                          39.93
Median TTFT (ms):                        41.92
P99 TTFT (ms):                           61.96
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.34
Median TPOT (ms):                        3.17
P99 TPOT (ms):                           5.61
---------------Inter-token Latency----------------
Mean ITL (ms):                           10.54
Median ITL (ms):                         9.84
P99 ITL (ms):                            35.23
==================================================

# BS=8
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             8
Benchmark duration (s):                  13.46
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              7.43
Output token throughput (tok/s):         1638.82
Peak output token throughput (tok/s):    655.00
Peak concurrent requests:                19.00
Total Token throughput (tok/s):          3366.71
---------------Time to First Token----------------
Mean TTFT (ms):                          42.27
Median TTFT (ms):                        43.26
P99 TTFT (ms):                           93.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.82
Median TPOT (ms):                        3.94
P99 TPOT (ms):                           21.36
---------------Inter-token Latency----------------
Mean ITL (ms):                           13.06
Median ITL (ms):                         10.94
P99 ITL (ms):                            37.37
==================================================