Перейти к содержанию

Оптимизация пропускной способности GPT-OSS-20B на NVIDIA H100 GPU

Вывод

gpt-oss-20b-h100

Рекомендуемая конфигурация для оптимизации пропускной способности GPT-OSS-20B на H100 GPU:

Команда обслуживания модели 
1
vllm serve openai/gpt-oss-20b --async-scheduling

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка базовая конфигурация (vLLM без оптимизации) После оптимизации
ShareGPT Общий TPS: 16701.31
Среднее TPOT(ms): 91.33		 Общий TPS: 19128.79 (+14.5%)
Среднее TPOT(ms): 86.81
Короткий запрос Общий TPS: 28032.23
Среднее TPOT(ms): 178.91		 Общий TPS: 32611.91 (+16.3%)
Среднее TPOT(ms): 75.14
Запрос средней длины Общий TPS: 32113.65
Среднее TPOT(ms): 149.76		 Общий TPS: 33149.54 (+3.2%)
Среднее TPOT(ms): 146.63
Длинный запрос Общий TPS: 33236.17
Среднее TPOT(ms): 184.26		 Общий TPS: 33823.34 (+1.8%)
Среднее TPOT(ms): 182.69
Очень длинный запрос Общий TPS: 29730.30
Среднее TPOT(ms): 239.44		 Общий TPS: 29718.21 (-0.0%)
Среднее TPOT(ms): 239.49

Примечание

  1. Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
  2. Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph, Torch Compile и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
  3. Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Цель оптимизации

Высокая пропускная способность при высокой конкурентности запросов.

Условия эксперимента

Модель

GPT-OSS-20B

Оборудование

Один NVIDIA H100 SXM GPU.

Версия движка

  • vLLM: v0.11.0
  • SGLang: v0.5.4.post2
  • TensorRT-LLM: v1.2.0rc1

Набор данных бенчмарка

  1. ShareGPT
  2. Случайный набор данных с различной длиной последовательностей:
    • Очень длинный запрос: 32000 входных токенов, 100 выходных токенов
    • Длинный запрос: 4000 входных токенов, 200 выходных токенов
    • Запрос средней длины: 2000 входных токенов, 100 выходных токенов
    • Короткий запрос: 128 входных токенов, 4 выходных токенов

Скрипт бенчмарка

Для оценки производительности модели используется vLLM bench CLI. Бенчмарк запускается следующей командой:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model openai/gpt-oss-20b --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model openai/gpt-oss-20b --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Результаты эксперимента

1. Выбор движка инференса

vLLM

Скрипт обслуживания модели 
1
vllm serve openai/gpt-oss-20b
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  24.81
Total input tokens:                      215312
Total generated tokens:                  193979
Request throughput (req/s):              40.30
Output token throughput (tok/s):         7817.03
Peak output token throughput (tok/s):    15953.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          16493.75
---------------Time to First Token----------------
Mean TTFT (ms):                          4881.62
Median TTFT (ms):                        4791.97
P99 TTFT (ms):                           8166.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          91.00
Median TPOT (ms):                        55.52
P99 TPOT (ms):                           237.48
---------------Inter-token Latency----------------
Mean ITL (ms):                           44.98
Median ITL (ms):                         31.92
P99 ITL (ms):                            248.25
==================================================

SGLang

Скрипт обслуживания модели 
1
python3 -m sglang.launch_server --model-path openai/gpt-oss-20b
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  73.05
Total input tokens:                      215312
Total generated tokens:                  195734
Request throughput (req/s):              13.69
Output token throughput (tok/s):         2679.49
Peak output token throughput (tok/s):    13804.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5626.99
---------------Time to First Token----------------
Mean TTFT (ms):                          10412.75
Median TTFT (ms):                        10326.68
P99 TTFT (ms):                           11920.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          314.53
Median TPOT (ms):                        215.95
P99 TPOT (ms):                           1052.78
---------------Inter-token Latency----------------
Mean ITL (ms):                           161.29
Median ITL (ms):                         47.78
P99 ITL (ms):                            1929.13
==================================================

TensorRT-LLM

Скрипт обслуживания модели 
1
trtllm-serve openai/gpt-oss-20b
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  53.03
Total input tokens:                      215312
Total generated tokens:                  191726
Request throughput (req/s):              18.86
Output token throughput (tok/s):         3615.45
Peak output token throughput (tok/s):    18641.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          7675.67
---------------Time to First Token----------------
Mean TTFT (ms):                          12248.87
Median TTFT (ms):                        12024.20
P99 TTFT (ms):                           22142.67
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          224.49
Median TPOT (ms):                        134.32
P99 TPOT (ms):                           552.52
---------------Inter-token Latency----------------
Mean ITL (ms):                           51.52
Median ITL (ms):                         0.07
P99 ITL (ms):                            963.17
==================================================

Результат: vLLM (16493.75 tok/s) > TensorRT-LLM (7675.67 tok/s) > SGLang(5626.99 tok/s)

2. Асинхронное планирование в vLLM

Скрипт обслуживания модели 
1
vllm serve openai/gpt-oss-20b --async-scheduling
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  21.39
Total input tokens:                      215312
Total generated tokens:                  193919
Request throughput (req/s):              46.74
Output token throughput (tok/s):         9064.40
Peak output token throughput (tok/s):    22786.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          19128.79
---------------Time to First Token----------------
Mean TTFT (ms):                          4933.91
Median TTFT (ms):                        4761.74
P99 TTFT (ms):                           8029.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          86.81
Median TPOT (ms):                        52.23
P99 TPOT (ms):                           222.47
---------------Inter-token Latency----------------
Mean ITL (ms):                           38.92
Median ITL (ms):                         27.02
P99 ITL (ms):                            236.29
==================================================

5. Настройка числа токенов в батче в vLLM

Скрипт обслуживания модели 
1
vllm serve openai/gpt-oss-120b --async-scheduling --max-num-batched-tokens 16384
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  21.56
Total input tokens:                      215312
Total generated tokens:                  194591
Request throughput (req/s):              46.39
Output token throughput (tok/s):         9027.14
Peak output token throughput (tok/s):    21433.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          19015.54
---------------Time to First Token----------------
Mean TTFT (ms):                          4957.11
Median TTFT (ms):                        4698.76
P99 TTFT (ms):                           7875.07
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          104.05
Median TPOT (ms):                        46.28
P99 TPOT (ms):                           443.08
---------------Inter-token Latency----------------
Mean ITL (ms):                           38.42
Median ITL (ms):                         24.01
P99 ITL (ms):                            448.17
==================================================

Сводка методов оптимизации

Метод оптимизации Прирост пропускной способности
Выбор движка -
Асинхронное планирование +16.0%
Токены в батче -

Другие сценарии бенчмарка

Мы дополнительно проверили оптимизированную конфигурацию на разных нагрузках, чтобы оценить её обобщаемость.

Базовый скрипт обслуживания модели 
1
vllm serve openai/gpt-oss-20b
Результаты базового бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  107.95
Total input tokens:                      3200000
Total generated tokens:                  9350
Request throughput (req/s):              0.93
Output token throughput (tok/s):         86.62
Total Token throughput (tok/s):          29730.30
---------------Time to First Token----------------
Mean TTFT (ms):                          53687.69
Median TTFT (ms):                        53539.25
P99 TTFT (ms):                           105655.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          239.44
Median TPOT (ms):                        273.86
P99 TPOT (ms):                           275.66
---------------Inter-token Latency----------------
Mean ITL (ms):                           247.74
Median ITL (ms):                         266.37
P99 ITL (ms):                            513.87
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  61.73
Total input tokens:                      1999249
Total generated tokens:                  52309
Request throughput (req/s):              8.10
Output token throughput (tok/s):         847.43
Total Token throughput (tok/s):          33236.17
---------------Time to First Token----------------
Mean TTFT (ms):                          31245.12
Median TTFT (ms):                        30820.97
P99 TTFT (ms):                           58702.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          184.26
Median TPOT (ms):                        225.55
P99 TPOT (ms):                           230.22
---------------Inter-token Latency----------------
Mean ITL (ms):                           192.92
Median ITL (ms):                         227.42
P99 ITL (ms):                            458.94
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  32.58
Total input tokens:                      999120
Total generated tokens:                  47022
Request throughput (req/s):              15.35
Output token throughput (tok/s):         1443.44
Total Token throughput (tok/s):          32113.65
---------------Time to First Token----------------
Mean TTFT (ms):                          16515.86
Median TTFT (ms):                        16363.96
P99 TTFT (ms):                           30359.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          149.76
Median TPOT (ms):                        159.20
P99 TPOT (ms):                           225.79
---------------Inter-token Latency----------------
Mean ITL (ms):                           159.21
Median ITL (ms):                         220.69
P99 ITL (ms):                            450.04
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  4.70
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              212.76
Output token throughput (tok/s):         851.04
Total Token throughput (tok/s):          28032.23
---------------Time to First Token----------------
Mean TTFT (ms):                          3174.14
Median TTFT (ms):                        2892.24
P99 TTFT (ms):                           4415.25
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          178.91
Median TPOT (ms):                        208.03
P99 TPOT (ms):                           246.08
---------------Inter-token Latency----------------
Mean ITL (ms):                           274.82
Median ITL (ms):                         228.45
P99 ITL (ms):                            509.42
==================================================
Скрипт обслуживания после оптимизации 
1
vllm serve openai/gpt-oss-20b --async-scheduling
Результаты бенчмарка после оптимизации 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  107.99
Total input tokens:                      3200000
Total generated tokens:                  9404
Request throughput (req/s):              0.93
Output token throughput (tok/s):         87.08
Peak output token throughput (tok/s):    1196.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          29718.21
---------------Time to First Token----------------
Mean TTFT (ms):                          53565.41
Median TTFT (ms):                        53436.03
P99 TTFT (ms):                           105793.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          239.49
Median TPOT (ms):                        275.30
P99 TPOT (ms):                           276.86
---------------Inter-token Latency----------------
Mean ITL (ms):                           247.68
Median ITL (ms):                         269.91
P99 ITL (ms):                            515.55
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  60.67
Total input tokens:                      1999249
Total generated tokens:                  52966
Request throughput (req/s):              8.24
Output token throughput (tok/s):         872.95
Peak output token throughput (tok/s):    7931.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          33823.34
---------------Time to First Token----------------
Mean TTFT (ms):                          30513.46
Median TTFT (ms):                        30091.29
P99 TTFT (ms):                           58206.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          182.69
Median TPOT (ms):                        224.54
P99 TPOT (ms):                           229.32
---------------Inter-token Latency----------------
Mean ITL (ms):                           191.65
Median ITL (ms):                         226.01
P99 ITL (ms):                            458.93
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  31.55
Total input tokens:                      999120
Total generated tokens:                  46814
Request throughput (req/s):              15.85
Output token throughput (tok/s):         1483.71
Peak output token throughput (tok/s):    12476.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          33149.54
---------------Time to First Token----------------
Mean TTFT (ms):                          16044.42
Median TTFT (ms):                        16014.33
P99 TTFT (ms):                           29736.08
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          146.63
Median TPOT (ms):                        154.39
P99 TPOT (ms):                           223.69
---------------Inter-token Latency----------------
Mean ITL (ms):                           155.89
Median ITL (ms):                         216.15
P99 ITL (ms):                            447.46
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  4.04
Total input tokens:                      127755
Total generated tokens:                  4000
Request throughput (req/s):              247.52
Output token throughput (tok/s):         990.08
Peak output token throughput (tok/s):    1364.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          32611.91
---------------Time to First Token----------------
Mean TTFT (ms):                          2884.78
Median TTFT (ms):                        2641.45
P99 TTFT (ms):                           3879.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          75.14
Median TPOT (ms):                        16.98
P99 TPOT (ms):                           221.60
---------------Inter-token Latency----------------
Mean ITL (ms):                           156.10
Median ITL (ms):                         151.36
P99 ITL (ms):                            452.14
==================================================