Перейти к содержанию

Оптимизация пропускной способности Qwen3-8B на ASCEND 910B NPU

Вывод

qwen3-8b-910b

Рекомендуемая конфигурация для оптимизации пропускной способности Qwen3-8B на ASCEND 910B NPU:

Команда обслуживания модели 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching

Сравнение результатов бенчмарков до и после оптимизации:

Сценарий бенчмарка базовая конфигурация (vLLM без оптимизации) После оптимизации
ShareGPT Общий TPS: 3143.94
Среднее TPOT(ms): 27.61		 Общий TPS: 6256.39 (+99.0%)
Среднее TPOT(ms): 796.13
Короткий запрос Общий TPS: 5834.79
Среднее TPOT(ms): 125.55		 Общий TPS: 16611.86 (+184.7%)
Среднее TPOT(ms): 1131.26
Запрос средней длины Общий TPS: 9589.00
Среднее TPOT(ms): 140.39		 Общий TPS: 14484.86 (+51.1%)
Среднее TPOT(ms): 122.59
Длинный запрос Общий TPS: 7677.41
Среднее TPOT(ms): 99.26		 Общий TPS: 11159.01 (+45.4%)
Среднее TPOT(ms): 97.74
Очень длинный запрос Общий TPS: 2890.64
Среднее TPOT(ms): 344.18		 Общий TPS: 7835.01 (+171.1%)
Среднее TPOT(ms): 169.70

Примечание

  1. Наши бенчмарки не охватывают все возможные сочетания оптимизаций. Например, отправной точкой дальнейшей настройки становится движок инференса с лучшей производительностью при настройках по умолчанию. Такое отсечение вариантов даёт локальный оптимум, который может не совпадать с глобальным.
  2. Есть и другие методы оптимизации, зависящие от сценария использования: максимальный размер батча, настройки планировщика, расширенный KV-кэш, CUDA graph, Torch Compile и т. д. Выводы этого документа могут служить отправной точкой для более целенаправленной оптимизации.
  3. Тесты проводились на определённых аппаратных и программных конфигурациях. Развитие движков инференса может привести к новым выводам.

Если что-то упущено или требуется обновление с учётом новых изменений, сообщите нам.

Цель оптимизации

Высокая пропускная способность при высокой конкурентности запросов.

Условия эксперимента

Модель

Qwen/Qwen3-30B-A3B

Оборудование

Один Ascend 910B NPU.

Версия движка

  • vLLM-Ascend: v0.9.1
  • MindIE: 2.1RC1

Набор данных бенчмарка

  1. ShareGPT
  2. Случайный набор данных с различной длиной последовательностей:
    • Очень длинный запрос: 32000 входных токенов, 100 выходных токенов
    • Длинный запрос: 4000 входных токенов, 200 выходных токенов
    • Запрос средней длины: 2000 входных токенов, 100 выходных токенов
    • Короткий запрос: 128 входных токенов, 4 выходных токенов

Скрипт бенчмарка

Для оценки производительности модели используется vLLM bench CLI. Бенчмарк запускается следующей командой:

1
2
3
4
5
6
7
8
# Prepare the ShareGPT dataset
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

# Benchmark on ShareGPT dataset
vllm bench serve --model Qwen/Qwen3-8B --backend openai-chat --endpoint /v1/chat/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

# Benchmark on random dataset (fixed seed for reproducibility)
vllm bench serve --model Qwen/Qwen3-8B --backend openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 4000 --random-output-len 200 --num-prompts 500 --seed 42

Результаты эксперимента

1. Выбор движка инференса

vLLM

Скрипт обслуживания модели 
1
vllm serve Qwen/Qwen3-8B
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  133.23
Total input tokens:                      217393
Total generated tokens:                  201469
Request throughput (req/s):              7.51
Output token throughput (tok/s):         1512.21
Peak output token throughput (tok/s):    2821.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          3143.94
---------------Time to First Token----------------
Mean TTFT (ms):                          43963.16
Median TTFT (ms):                        41177.19
P99 TTFT (ms):                           91086.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.09
Median TPOT (ms):                        122.07
P99 TPOT (ms):                           269.65
---------------Inter-token Latency----------------
Mean ITL (ms):                           112.90
Median ITL (ms):                         78.96
P99 ITL (ms):                            280.07
==================================================

MindIE

Скрипт обслуживания модели 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve Qwen/Qwen3-8B
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  74.36
Total input tokens:                      217393
Total generated tokens:                  201609
Request throughput (req/s):              13.45
Output token throughput (tok/s):         2711.10
Peak output token throughput (tok/s):    4588.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5634.45
---------------Time to First Token----------------
Mean TTFT (ms):                          8242.22
Median TTFT (ms):                        8298.00
P99 TTFT (ms):                           18074.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          788.31
Median TPOT (ms):                        204.41
P99 TPOT (ms):                           6547.97
---------------Inter-token Latency----------------
Mean ITL (ms):                           155.96
Median ITL (ms):                         45.35
P99 ITL (ms):                            218.57
==================================================

Результат: MineIE (5634.4 tok/s) > vLLM (3143.94 tok/s)

2. Quantization в MindIE

Скрипт обслуживания модели 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  69.32
Total input tokens:                      217393
Total generated tokens:                  201315
Request throughput (req/s):              14.42
Output token throughput (tok/s):         2903.93
Peak output token throughput (tok/s):    4799.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6039.79
---------------Time to First Token----------------
Mean TTFT (ms):                          5834.61
Median TTFT (ms):                        5648.87
P99 TTFT (ms):                           11181.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          822.87
Median TPOT (ms):                        188.93
P99 TPOT (ms):                           7710.20
---------------Inter-token Latency----------------
Mean ITL (ms):                           148.97
Median ITL (ms):                         45.31
P99 ITL (ms):                            97.87
==================================================

3. Включение кэша префиксов в MindIE

Скрипт обслуживания модели 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  66.96
Total input tokens:                      217393
Total generated tokens:                  201517
Request throughput (req/s):              14.93
Output token throughput (tok/s):         3009.64
Peak output token throughput (tok/s):    4985.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6256.39
---------------Time to First Token----------------
Mean TTFT (ms):                          5320.32
Median TTFT (ms):                        5159.10
P99 TTFT (ms):                           10384.83
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          796.13
Median TPOT (ms):                        183.35
P99 TPOT (ms):                           7267.96
---------------Inter-token Latency----------------
Mean ITL (ms):                           141.91
Median ITL (ms):                         43.55
P99 ITL (ms):                            115.20
==================================================

4. Настройка размера батча в MindIE

Скрипт обслуживания модели 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching --max-batch-size=400 --max-prefill-batch-size=200
Результат бенчмарка 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  69.07
Total input tokens:                      217393
Total generated tokens:                  201424
Request throughput (req/s):              14.48
Output token throughput (tok/s):         2916.34
Peak output token throughput (tok/s):    6017.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          6063.89
---------------Time to First Token----------------
Mean TTFT (ms):                          6335.34
Median TTFT (ms):                        5812.45
P99 TTFT (ms):                           11260.65
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          581.93
Median TPOT (ms):                        167.14
P99 TPOT (ms):                           5501.66
---------------Inter-token Latency----------------
Mean ITL (ms):                           143.92
Median ITL (ms):                         70.27
P99 ITL (ms):                            281.16
==================================================

Сводка методов оптимизации

Метод оптимизации Прирост пропускной способности
Выбор движка +79.2%
Quantization +7.2%
Кэширование префиксов +3.6%
Настройка размера батча -

Другие сценарии бенчмарка

Мы дополнительно проверили оптимизированную конфигурацию на разных нагрузках, чтобы оценить её обобщаемость.

Базовый скрипт обслуживания модели 
1
vllm serve Qwen/Qwen3-8B
Результаты базового бенчмарка 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  1110.48
Total input tokens:                      3200000
Total generated tokens:                  10000
Request throughput (req/s):              0.09
Output token throughput (tok/s):         9.01
Peak output token throughput (tok/s):    147.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          2890.64
---------------Time to First Token----------------
Mean TTFT (ms):                          559550.01
Median TTFT (ms):                        561560.86
P99 TTFT (ms):                           1096851.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          344.18
Median TPOT (ms):                        346.89
P99 TPOT (ms):                           668.43
---------------Inter-token Latency----------------
Mean ITL (ms):                           340.74
Median ITL (ms):                         48.54
P99 ITL (ms):                            59.04
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  273.17
Total input tokens:                      1997463
Total generated tokens:                  99812
Request throughput (req/s):              1.83
Output token throughput (tok/s):         365.38
Peak output token throughput (tok/s):    1062.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          7677.41
---------------Time to First Token----------------
Mean TTFT (ms):                          132541.20
Median TTFT (ms):                        135284.67
P99 TTFT (ms):                           262452.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          99.26
Median TPOT (ms):                        100.01
P99 TPOT (ms):                           175.24
---------------Inter-token Latency----------------
Mean ITL (ms):                           98.69
Median ITL (ms):                         55.84
P99 ITL (ms):                            134.52
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  109.30
Total input tokens:                      998057
Total generated tokens:                  50000
Request throughput (req/s):              4.57
Output token throughput (tok/s):         457.47
Peak output token throughput (tok/s):    1904.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          9589.00
---------------Time to First Token----------------
Mean TTFT (ms):                          53978.52
Median TTFT (ms):                        57661.39
P99 TTFT (ms):                           103428.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          140.39
Median TPOT (ms):                        142.80
P99 TPOT (ms):                           275.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           139.05
Median ITL (ms):                         63.53
P99 ITL (ms):                            627.10
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  22.59
Total input tokens:                      127782
Total generated tokens:                  4000
Request throughput (req/s):              44.28
Output token throughput (tok/s):         177.10
Peak output token throughput (tok/s):    1280.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          5834.79
---------------Time to First Token----------------
Mean TTFT (ms):                          18464.39
Median TTFT (ms):                        17489.55
P99 TTFT (ms):                           22134.55
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.55
Median TPOT (ms):                        135.76
P99 TPOT (ms):                           149.27
---------------Inter-token Latency----------------
Mean ITL (ms):                           94.16
Median ITL (ms):                         83.83
P99 ITL (ms):                            251.91
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     100
Maximum request concurrency:             4
Benchmark duration (s):                  275.57
Total input tokens:                      23260
Total generated tokens:                  22061
Request throughput (req/s):              0.36
Output token throughput (tok/s):         80.06
Peak output token throughput (tok/s):    92.00
Peak concurrent requests:                8.00
Total Token throughput (tok/s):          164.46
---------------Time to First Token----------------
Mean TTFT (ms):                          145.88
Median TTFT (ms):                        142.78
P99 TTFT (ms):                           201.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          47.28
Median TPOT (ms):                        47.29
P99 TPOT (ms):                           50.41
---------------Inter-token Latency----------------
Mean ITL (ms):                           47.10
Median ITL (ms):                         46.72
P99 ITL (ms):                            91.82
==================================================
Скрипт обслуживания после оптимизации 
1
2
3
# This is a simplified vLLM-like command. GPUStack maps arguments to the
# corresponding MindIE JSON configuration and runs with mindieservice_daemon.
mindie serve vllm-ascend/Qwen3-8B-W8A8 --enable-prefix-caching
Результаты бенчмарка после оптимизации 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# random 32K input
============ Serving Benchmark Result ============
Successful requests:                     100
Benchmark duration (s):                  296.36
Total input tokens:                      3200000
Total generated tokens:                  9693
Request throughput (req/s):              0.34
Output token throughput (tok/s):         32.71
Peak output token throughput (tok/s):    330.00
Peak concurrent requests:                100.00
Total Token throughput (tok/s):          10830.52
---------------Time to First Token----------------
Mean TTFT (ms):                          147133.58
Median TTFT (ms):                        147851.57
P99 TTFT (ms):                           289974.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          242.08
Median TPOT (ms):                        232.74
P99 TPOT (ms):                           741.49
---------------Inter-token Latency----------------
Mean ITL (ms):                           225.71
Median ITL (ms):                         46.46
P99 ITL (ms):                            56.81
==================================================

# random 4K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  157.15
Total input tokens:                      2000000
Total generated tokens:                  99961
Request throughput (req/s):              3.18
Output token throughput (tok/s):         636.07
Peak output token throughput (tok/s):    2142.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          13362.49
---------------Time to First Token----------------
Mean TTFT (ms):                          70716.71
Median TTFT (ms):                        74982.88
P99 TTFT (ms):                           148427.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          130.97
Median TPOT (ms):                        134.98
P99 TPOT (ms):                           208.16
---------------Inter-token Latency----------------
Mean ITL (ms):                           130.95
Median ITL (ms):                         63.84
P99 ITL (ms):                            489.52
==================================================

# random 2K input
============ Serving Benchmark Result ============
Successful requests:                     500
Benchmark duration (s):                  67.34
Total input tokens:                      1000000
Total generated tokens:                  49975
Request throughput (req/s):              7.42
Output token throughput (tok/s):         742.11
Peak output token throughput (tok/s):    3000.00
Peak concurrent requests:                500.00
Total Token throughput (tok/s):          15591.71
---------------Time to First Token----------------
Mean TTFT (ms):                          29078.68
Median TTFT (ms):                        32203.66
P99 TTFT (ms):                           62077.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          194.54
Median TPOT (ms):                        192.07
P99 TPOT (ms):                           358.38
---------------Inter-token Latency----------------
Mean ITL (ms):                           194.46
Median ITL (ms):                         68.54
P99 ITL (ms):                            4496.71
==================================================

# random 128 input
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  8.78
Total input tokens:                      128000
Total generated tokens:                  4000
Request throughput (req/s):              113.94
Output token throughput (tok/s):         455.76
Peak output token throughput (tok/s):    2511.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          15040.15
---------------Time to First Token----------------
Mean TTFT (ms):                          4491.88
Median TTFT (ms):                        4467.47
P99 TTFT (ms):                           7693.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1249.72
Median TPOT (ms):                        1258.92
P99 TPOT (ms):                           2252.88
---------------Inter-token Latency----------------
Mean ITL (ms):                           1249.72
Median ITL (ms):                         54.67
P99 ITL (ms):                            6378.56
==================================================

# ShareGPT batch size 4
============ Serving Benchmark Result ============
Successful requests:                     1000
Maximum request concurrency:             4
Benchmark duration (s):                  1292.33
Total input tokens:                      217393
Total generated tokens:                  201773
Request throughput (req/s):              0.77
Output token throughput (tok/s):         156.13
Peak output token throughput (tok/s):    200.00
Peak concurrent requests:                9.00
Total Token throughput (tok/s):          324.35
---------------Time to First Token----------------
Mean TTFT (ms):                          112.38
Median TTFT (ms):                        111.05
P99 TTFT (ms):                           154.06
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.42
Median TPOT (ms):                        24.47
P99 TPOT (ms):                           35.05
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.08
Median ITL (ms):                         23.05
P99 ITL (ms):                            79.54
==================================================