KV cache management
6 methods in the atlas attack this one problem. They are rivals: each wins something the others do not.
Phrasings that mean this problem
KV cache managementKV cache eviction
llm-inference
- PagedAttentionBlock-table cache pagingcanonllm-inference
- Prefix cachingShared-prompt cache reusestandardllm-inference
- KV cache quantizationPer-channel low-bit cachespecialistllm-inference
- StreamingLLMAttention-sink retentionspecialistllm-inference
- H2O cache evictionHeavy-hitter token scoringspecialistllm-inference
- SnapKVAttention-score poolingspecialistllm-inference