跳转至

NVIDIA A100 GPUs 上的 GLM-4.x 吞吐量优化

结论

GLM-4.x (4.5、4.6 和 4.7) 在 A100 GPUs 上的吞吐量优化推荐配置：

推理服务命令 
1
vllm serve zai-org/GLM-4.6-FP8 -tp 8

说明

  1. 基准测试未覆盖所有可能的优化组合。例如，我们会选择默认配置下表现最好的推理引擎作为后续调优的起点。这种剪枝方式得到的是局部最优解，未必是全局最优解。
  2. 其他优化方法取决于具体使用场景，包括最大批大小、调度配置、扩展 KV cache、CUDA graph 等。本文结论可作为进一步针对性优化的起点。
  3. 测试基于特定的硬件和软件环境。推理引擎的改进可能带来新的结论。
  4. 量化可能影响准确率，但对于大多数模型，FP8 量化的准确率下降可控制在 1% 以内。详情参见评估结果。因此，强烈建议在高吞吐量推理服务场景中使用 FP8 量化。

如有遗漏或需要根据最新变化更新的内容，请告诉我们。

优化目标

在高并发请求场景下实现高吞吐量。

实验环境

模型

zai-org/GLM-4.6

硬件

单节点配备 8 × NVIDIA A100 SXM 80GB GPU。

引擎版本

  • vLLM: v0.11.0
  • SGLang: v0.5.3
  • TensorRT-LLM: v1.0.0

基准测试数据集

  1. ShareGPT
  2. 不同序列长度的随机数据集：
    • 超长提示词：32000 输入 tokens，100 输出 tokens
    • 长提示词：4000 输入 tokens，200 输出 tokens
    • 中等长度提示词：2000 输入 tokens，100 输出 tokens
    • 短提示词：128 输入 tokens，4 输出 tokens

基准测试脚本

使用 vLLM bench CLI 工具对模型性能进行基准测试。运行基准测试的命令如下：

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model zai-org/GLM-4.6 --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model zai-org/GLM-4.6 --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

实验结果

受显存容量限制，原始 BF16 精度模型无法在配备 NVIDIA A100 GPU 的单台服务器上提供推理服务。因此，本测试重点评估 FP8 量化模型。

1. 选择推理引擎

vLLM

推理服务脚本 
1
vllm serve zai-org/GLM-4.6-FP8 -tp 8
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  148.61
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              6.73
Output token throughput (tok/s):         1345.12
Peak output token throughput (tok/s):    2500.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2788.21
---------------Time to First Token----------------
Mean TTFT (ms):                          51022.80
Median TTFT (ms):                        48308.20
P99 TTFT (ms):                           106768.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          154.20
Median TPOT (ms):                        144.84
P99 TPOT (ms):                           318.65
---------------Inter-token Latency----------------
Mean ITL (ms):                           134.46
Median ITL (ms):                         117.84
P99 ITL (ms):                            326.23
==================================================

SGLang

推理服务脚本 
1
python3 -m sglang.launch_server --model-path zai-org/GLM-4.6-FP8 --host 0.0.0.0 --port 8000 --tp-size 8
基准测试结果 
1
ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")

结果：vLLM (2788.21 tok/s) > SGLang(不支持) = TensorRT-LLM (不支持)

2. vLLM 中的 并行策略

TP+EP

推理服务脚本 
1
vllm serve zai-org/GLM-4.6-FP8 -tp 8 --enable-expert-parallel
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  153.11
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              6.53
Output token throughput (tok/s):         1305.62
Peak output token throughput (tok/s):    2311.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2706.34
---------------Time to First Token----------------
Mean TTFT (ms):                          51031.51
Median TTFT (ms):                        48265.76
P99 TTFT (ms):                           108400.10
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          158.11
Median TPOT (ms):                        149.37
P99 TPOT (ms):                           315.61
---------------Inter-token Latency----------------
Mean ITL (ms):                           139.37
Median ITL (ms):                         127.75
P99 ITL (ms):                            321.92
==================================================

3. vLLM 中的 最大批处理 Token 数

推理服务脚本 
1
vllm serve zai-org/GLM-4.6-FP8 -tp 8 --max-num-batched-tokens 8192
基准测试结果 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
# --max-num-batched-tokens 4096
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  146.98
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              6.80
Output token throughput (tok/s):         1360.05
Peak output token throughput (tok/s):    2502.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2819.17
---------------Time to First Token----------------
Mean TTFT (ms):                          49578.97
Median TTFT (ms):                        47246.86
P99 TTFT (ms):                           104952.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          159.13
Median TPOT (ms):                        144.06
P99 TPOT (ms):                           547.87
---------------Inter-token Latency----------------
Mean ITL (ms):                           134.40
Median ITL (ms):                         117.57
P99 ITL (ms):                            458.51
==================================================

# --max-num-batched-tokens 8192
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  146.69
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              6.82
Output token throughput (tok/s):         1362.78
Peak output token throughput (tok/s):    2385.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2824.82
---------------Time to First Token----------------
Mean TTFT (ms):                          49591.68
Median TTFT (ms):                        47020.36
P99 TTFT (ms):                           104829.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          160.28
Median TPOT (ms):                        143.24
P99 TPOT (ms):                           559.15
---------------Inter-token Latency----------------
Mean ITL (ms):                           134.08
Median ITL (ms):                         118.83
P99 ITL (ms):                            320.13
==================================================

# --max-num-batched-tokens 16384
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  147.02
Total input tokens:                      214465
Total generated tokens:                  199904
Request throughput (req/s):              6.80
Output token throughput (tok/s):         1359.74
Peak output token throughput (tok/s):    2435.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          2818.53
---------------Time to First Token----------------
Mean TTFT (ms):                          49736.96
Median TTFT (ms):                        47034.75
P99 TTFT (ms):                           105084.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          160.50
Median TPOT (ms):                        143.84
P99 TPOT (ms):                           582.59
---------------Inter-token Latency----------------
Mean ITL (ms):                           134.22
Median ITL (ms):                         118.12
P99 ITL (ms):                            325.43
==================================================

优化选项汇总

优化选项 吞吐量提升
引擎选择 -
并行策略 -
最大批处理 Token 数 +1.3%