Перейти к содержанию

Оптимизация пропускной способности GPT-OSS-20B на NVIDIA A100 GPU

Вывод

gpt-oss-20b-a100

Рекомендуемая конфигурация для оптимизации пропускной способности GPT-OSS-20B на A100 GPU:

Команда обслуживания модели 
1
vllm serve openai/gpt-oss-20b --async-scheduling

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка базовая конфигурация (vLLM без оптимизации) После оптимизации
ShareGPT Общий TPS: 9743.28
Среднее TPOT(ms): 48.57		 Общий TPS: 10919.92 (+12.1%)
Среднее TPOT(ms): 43.25
Короткий запрос Общий TPS: 24223.22
Среднее TPOT(ms): 62.10		 Общий TPS: 25172.82 (+4.0%)
Среднее TPOT(ms): 55.31
Запрос средней длины Общий TPS: 18021.34
Среднее TPOT(ms): 100.32		 Общий TPS: 18920.16 (+5.0%)
Среднее TPOT(ms): 95.80
Длинный запрос Общий TPS: 15938.87
Среднее TPOT(ms): 123.35		 Общий TPS: 16550.81 (+3.8%)
Среднее TPOT(ms): 119.37
Очень длинный запрос Общий TPS: 7280.33
Среднее TPOT(ms): 268.03		 Общий TPS: 7363.09 (+1.1%)
Среднее TPOT(ms): 264.93

Примечание

  1. Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
  2. Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph, Torch Compile и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
  3. Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Цель оптимизации

Высокая пропускная способность при высокой конкурентности запросов.

Условия эксперимента

Модель

GPT-OSS-20B

Оборудование

Один NVIDIA A100 SXM 80GB GPU.

Версия движка

  • vLLM: v0.11.0
  • SGLang: v0.5.4.post2
  • TensorRT-LLM: v1.2.0rc1

Набор данных бенчмарка

  1. ShareGPT
  2. Случайный набор данных с различной длиной последовательностей:
    • Очень длинный запрос: 32000 входных токенов, 100 выходных токенов
    • Длинный запрос: 4000 входных токенов, 200 выходных токенов
    • Запрос средней длины: 2000 входных токенов, 100 выходных токенов
    • Короткий запрос: 128 входных токенов, 4 выходных токенов

Скрипт бенчмарка

Для оценки производительности модели используется vLLM bench CLI. Бенчмарк запускается следующей командой:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model openai/gpt-oss-20b --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model openai/gpt-oss-20b --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Результаты эксперимента

1. Выбор движка инференса

vLLM

Скрипт обслуживания модели 
1
vllm serve openai/gpt-oss-20b
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  42.03
Total input tokens:                      215312
Total generated tokens:                  194171
Request throughput (req/s):              23.79
Output token throughput (tok/s):         4620.12
Peak output token throughput (tok/s):    7484.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          9743.28
---------------Time to First Token----------------
Mean TTFT (ms):                          13654.59
Median TTFT (ms):                        13063.93
P99 TTFT (ms):                           30306.72
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          48.57
Median TPOT (ms):                        45.15
P99 TPOT (ms):                           110.58
---------------Inter-token Latency----------------
Mean ITL (ms):                           41.95
Median ITL (ms):                         33.96
P99 ITL (ms):                            112.27
==================================================

SGLang

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path openai/gpt-oss-20b
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  735.43
Total input tokens:                      215312
Total generated tokens:                  195617
Request throughput (req/s):              1.36
Output token throughput (tok/s):         265.99
Peak output token throughput (tok/s):    5848.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          558.76
---------------Time to First Token----------------
Mean TTFT (ms):                          140888.75
Median TTFT (ms):                        137990.65
P99 TTFT (ms):                           274130.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6243.98
Median TPOT (ms):                        2508.45
P99 TPOT (ms):                           46225.15
---------------Inter-token Latency----------------
Mean ITL (ms):                           1889.78
Median ITL (ms):                         1305.64
P99 ITL (ms):                            4248.44
==================================================

TensorRT-LLM

Скрипт обслуживания модели 
1
trtllm-serve openai/gpt-oss-120b --max_seq_len 32768
Результат бенчмарка 
1
NotImplementedError: WFP4A16 MoE is unsupported on SM80.

Результат: vLLM (9743.28 tok/s) > SGLang(558.76 tok/s) > TensorRT-LLM (Не поддерживается)

2. Асинхронное планирование в vLLM

Скрипт обслуживания модели 
1
vllm serve openai/gpt-oss-20b --async-scheduling
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  37.51
Total input tokens:                      215312
Total generated tokens:                  194303
Request throughput (req/s):              26.66
Output token throughput (tok/s):         5179.92
Peak output token throughput (tok/s):    8545.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10919.92
---------------Time to First Token----------------
Mean TTFT (ms):                          12371.62
Median TTFT (ms):                        11647.98
P99 TTFT (ms):                           27040.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          43.25
Median TPOT (ms):                        39.47
P99 TPOT (ms):                           105.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           36.85
Median ITL (ms):                         28.41
P99 ITL (ms):                            106.78
==================================================

3. Настройка числа токенов в батче в vLLM

Скрипт обслуживания модели 
1
vllm serve openai/gpt-oss-20b --async-scheduling --max-num-batched-tokens 8192
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# --max-num-batched-tokens 4096
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  37.43
Total input tokens:                      215312
Total generated tokens:                  194145
Request throughput (req/s):              26.72
Output token throughput (tok/s):         5186.81
Peak output token throughput (tok/s):    8601.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10939.12
---------------Time to First Token----------------
Mean TTFT (ms):                          12298.27
Median TTFT (ms):                        11609.21
P99 TTFT (ms):                           27015.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          44.99
Median TPOT (ms):                        39.55
P99 TPOT (ms):                           195.62
---------------Inter-token Latency----------------
Mean ITL (ms):                           37.00
Median ITL (ms):                         28.57
P99 ITL (ms):                            118.90
==================================================
# --max-num-batched-tokens 8192
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  37.58
Total input tokens:                      215312
Total generated tokens:                  193986
Request throughput (req/s):              26.61
Output token throughput (tok/s):         5161.85
Peak output token throughput (tok/s):    8663.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          10891.17
---------------Time to First Token----------------
Mean TTFT (ms):                          12479.83
Median TTFT (ms):                        11828.63
P99 TTFT (ms):                           27160.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          45.94
Median TPOT (ms):                        39.39
P99 TPOT (ms):                           234.47
---------------Inter-token Latency----------------
Mean ITL (ms):                           36.95
Median ITL (ms):                         28.33
P99 ITL (ms):                            101.21
==================================================

Сводка методов оптимизации

Метод оптимизации Прирост пропускной способности
Выбор движка -
Асинхронное планирование +12.1%
Токены в батче -

Другие сценарии бенчмарка

Мы дополнительно проверили оптимизированную конфигурацию на разных нагрузках, чтобы оценить её обобщаемость.

Базовый скрипт обслуживания модели 
1
vllm serve openai/gpt-oss-20b
Результаты базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  440.85
Total input tokens:                      3200000
Total generated tokens:                  9533
Request throughput (req/s):              0.23
Output token throughput (tok/s):         21.62
Peak output token throughput (tok/s):    203.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          7280.33
---------------Time to First Token----------------
Mean TTFT (ms):                          219739.77
Median TTFT (ms):                        219663.48
P99 TTFT (ms):                           435080.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          268.03
Median TPOT (ms):                        278.17
P99 TPOT (ms):                           282.40
---------------Inter-token Latency----------------
Mean ITL (ms):                           276.43
Median ITL (ms):                         274.32
P99 ITL (ms):                            438.91
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  128.79
Total input tokens:                      1999249
Total generated tokens:                  53530
Request throughput (req/s):              3.88
Output token throughput (tok/s):         415.64
Peak output token throughput (tok/s):    1983.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          15938.87
---------------Time to First Token----------------
Mean TTFT (ms):                          62947.13
Median TTFT (ms):                        61941.78
P99 TTFT (ms):                           125429.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          123.35
Median TPOT (ms):                        129.79
P99 TPOT (ms):                           131.42
---------------Inter-token Latency----------------
Mean ITL (ms):                           130.61
Median ITL (ms):                         129.50
P99 ITL (ms):                            262.38
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  58.06
Total input tokens:                      999120
Total generated tokens:                  47224
Request throughput (req/s):              8.61
Output token throughput (tok/s):         813.35
Peak output token throughput (tok/s):    4701.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          18021.34
---------------Time to First Token----------------
Mean TTFT (ms):                          31669.55
Median TTFT (ms):                        32083.12
P99 TTFT (ms):                           55891.47
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          100.32
Median TPOT (ms):                        119.20
P99 TPOT (ms):                           119.71
---------------Inter-token Latency----------------
Mean ITL (ms):                           106.87
Median ITL (ms):                         119.09
P99 ITL (ms):                            239.78
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  5.44
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              183.85
Output token throughput (tok/s):         735.40
Peak output token throughput (tok/s):    932.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          24223.22
---------------Time to First Token----------------
Mean TTFT (ms):                          2774.46
Median TTFT (ms):                        1971.28
P99 TTFT (ms):                           5298.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          62.10
Median TPOT (ms):                        65.44
P99 TPOT (ms):                           106.62
---------------Inter-token Latency----------------
Mean ITL (ms):                           100.20
Median ITL (ms):                         105.43
P99 ITL (ms):                            213.73
==================================================
Скрипт обслуживания после оптимизации 
1
vllm serve openai/gpt-oss-20b --async-scheduling
Результаты бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  435.88
Total input tokens:                      3200000
Total generated tokens:                  9394
Request throughput (req/s):              0.23
Output token throughput (tok/s):         21.55
Peak output token throughput (tok/s):    239.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          7363.09
---------------Time to First Token----------------
Mean TTFT (ms):                          217408.40
Median TTFT (ms):                        217388.75
P99 TTFT (ms):                           430331.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          264.93
Median TPOT (ms):                        275.04
P99 TPOT (ms):                           278.94
---------------Inter-token Latency----------------
Mean ITL (ms):                           274.36
Median ITL (ms):                         274.48
P99 ITL (ms):                            433.63
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  123.99
Total input tokens:                      1999249
Total generated tokens:                  52857
Request throughput (req/s):              4.03
Output token throughput (tok/s):         426.31
Peak output token throughput (tok/s):    1699.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          16550.81
---------------Time to First Token----------------
Mean TTFT (ms):                          60587.64
Median TTFT (ms):                        59590.47
P99 TTFT (ms):                           121079.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          119.37
Median TPOT (ms):                        125.63
P99 TPOT (ms):                           126.57
---------------Inter-token Latency----------------
Mean ITL (ms):                           126.46
Median ITL (ms):                         125.29
P99 ITL (ms):                            253.57
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  55.29
Total input tokens:                      999120
Total generated tokens:                  46961
Request throughput (req/s):              9.04
Output token throughput (tok/s):         849.37
Peak output token throughput (tok/s):    4961.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          18920.16
---------------Time to First Token----------------
Mean TTFT (ms):                          30030.76
Median TTFT (ms):                        30306.12
P99 TTFT (ms):                           53186.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          95.80
Median TPOT (ms):                        114.20
P99 TPOT (ms):                           114.34
---------------Inter-token Latency----------------
Mean ITL (ms):                           102.24
Median ITL (ms):                         114.01
P99 ITL (ms):                            229.30
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  5.23
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              191.06
Output token throughput (tok/s):         764.23
Peak output token throughput (tok/s):    1302.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          25172.82
---------------Time to First Token----------------
Mean TTFT (ms):                          2682.01
Median TTFT (ms):                        1944.72
P99 TTFT (ms):                           5078.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          55.31
Median TPOT (ms):                        46.59
P99 TPOT (ms):                           102.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           85.84
Median ITL (ms):                         100.82
P99 ITL (ms):                            205.12
==================================================