Evaluating LMCache Prefill Acceleration in vLLM

LMCache is an extensible KV Cache Layer for LLM inference designed to address key challenges in large-scale deployment scenarios. This documentation evaluates the performance impact of LMCache on vLLM inference, particularly focusing on prefill stage acceleration and its implications for various workload patterns.

Conclusions

1. LMCache provides significant prefill acceleration in scenarios with high cache hit rates, achieving up to +355.3% input TPS improvement and -58.8% reduction in TTFT for long-context (20K tokens) multi-turn conversations in the experiments.

2. Performance benefits are highly workload-dependent: - Optimal scenarios: Multi-turn conversations with shared prefixes and repeated patterns - Suboptimal scenarios: Random inputs with no cache reuse patterns

3. Chunk size optimization The default 256 chunk size shows the optimal results in tested configurations.

4. Cache miss scenarios incur overhead, showing -3% to -15% performance degradation when no cache reuse occurs, making LMCache most suitable for workloads with predictable prefix patterns.

Technical Background

LMCache Overview

LMCache extends vLLM's KV cache capabilities through:

Key Use Cases

1. Low Prefix Cache Hit Rates: Mitigates GPU VRAM limitations and cache eviction issues
2. Distributed Cache Sharing: Enables cache sharing across multiple vLLM instances
3. PD Disaggregation: Supports disaggregated deployment architectures

Experimental Setup

• Model: Qwen3-8B
• Hardware: NVIDIA RTX 4090 24GB
• vLLM Version: v0.10.1.1
• Benchmark Method: Multi-turn conversation benchmark

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Standard vLLM serving
vllm serve Qwen/Qwen3-8B

# LMCache-enabled serving
##### lmcache_config.yaml
chunk_size: 256
local_cpu: true
max_local_cpu_size: 50
#####
LMCACHE_CONFIG_FILE=/root/lmcache_config.yaml vllm serve /root/Qwen3-8B \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# Multi-turn bench scripts
# Ref: https://github.com/vllm-project/vllm/tree/main/benchmarks/multi_turn

##### generate_multi_turn.json
{
    "filetype": "generate_conversations",
    "num_conversations": 24,
    "text_files": ["pg1184.txt"],
    "print_stats": false,
    "prompt_input": {
        "num_turns": {
            "distribution": "uniform",
            "min": 12,
            "max": 18
        },
        "common_prefix_num_tokens": {
            "distribution": "constant",
            "value": 500
        },
        "prefix_num_tokens": {
            "distribution": "lognormal",
            "average": 4000,
            "max": 20000
        },
        "num_tokens": {
            "distribution": "uniform",
            "min": 120,
            "max": 160
        }
    },
    "prompt_output": {
        "num_tokens": {
            "distribution": "uniform",
            "min": 80,
            "max": 120
        }
    }
}
#####

python benchmark_serving_multi_turn.py --model $MODEL_PATH --input-file generate_multi_turn.json --num-clients 10 --max-active-conversations 10

Experimental Results

Multi-turn Conversation Performance

5K Input Tokens

20K Input Tokens

20K Input Tokens + 1 Output Token

Tuning Chunk Size

Cache Miss Scenarios (Random Dataset)

1

vllm bench serve --model Qwen/Qwen3-8B --endpoint-type openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 1024 --random-output-len 128 --num-prompts 100 --seed 40

1K Input Tokens

8K Input Tokens

20K Input Tokens

All VRAM KV Cache Hit Scenarios

1K Input Tokens

8K Input Tokens

20K Input Tokens

Remote Storage Backend Performance (20K Tokens TTFT)