HybridKV: Query-Agnostic KV Cache Compression via Head-Dominant Context Reconstruction

ICMLw 2026
Changwoo Baek  ·  Kyeongbo Kong†
Pusan National University
Paper
Full vs hybrid context-reconstruction attention
Full vs. hybrid context-reconstruction attention. (a) Full reconstruction scores importance over all attention heads, incurring high compression overhead. (b) HybridKV pre-selects important heads in a context-independent way, then applies token-level reconstruction scoring only within them. (c) Overhead vs. KVzip and (d) accuracy under different KV cache ratios on Qwen2.5-7B-1M.
TL;DR

Reconstruction-based KV importance is head-dominant — it concentrates in a small, largely context-stable subset of heads. HybridKV pre-selects those heads offline and prunes tokens only within them, matching full reconstruction accuracy while cutting compression overhead by up to 36%.

Abstract

Efficient key–value (KV) cache compression is crucial for long-context large language models, but accurate query-agnostic compression based on context reconstruction incurs substantial additional prefill-time overhead. This letter observes that reconstruction-based KV importance is head-dominant: high-importance entries repeatedly concentrate in a small subset of attention heads that remains largely stable across contexts. Based on this observation, we propose HybridKV, a hybrid KV cache compression method that pre-selects salient heads offline and applies token-level reconstruction-based pruning only within the selected heads during deployment. By avoiding redundant reconstruction scoring over low-importance heads, HybridKV reduces the additional reconstruction-based compression overhead while preserving context-adaptive token selection. Experiments on long-context benchmarks show that HybridKV achieves accuracy comparable to full context-reconstruction compression and reduces compression-only overhead by up to 36%.

Contributions

Head-dominant structure

Reconstruction importance concentrates on a small, largely context-independent subset of heads (79.2% avg. Jaccard overlap of top-50% heads across 100 SQuAD contexts), while token-level variation persists within them.

HybridKV

Pre-select salient heads offline from a tiny calibration set, then restrict token-level reconstruction-based pruning to those heads at deployment — removing redundant head-level computation.

Up to 36% less overhead

Across long-context benchmarks and three LLMs, HybridKV matches full context-reconstruction accuracy while reducing compression-only overhead by up to 36%.

Reconstruction importance is head-dominant

Head-dominant patterns in context-reconstruction attention
Head-dominant patterns in context-reconstruction attention. Reconstruction importance is concentrated in a small subset of layer–head indices while most other heads stay consistently low, and the dominant head locations are largely preserved across contexts — the top-50% head sets from 100 SQuAD contexts show a 79.2% average pairwise Jaccard overlap. Nonetheless, meaningful token-level score variation persists within the salient heads.

Hybrid context-reconstruction compression

Offline head pre-selection → token-level reconstruction pruning within the retained heads.

1) Offline head pre-selection. We aggregate context-reconstruction attention scores over a tiny calibration set (10 contexts, ~10K tokens from LongAlpaca-12k). For each head we take the maximum score across samples, rank heads, and keep a fraction k — the head-retention ratio (equivalently, 1−k heads are pruned).

2) Deployment-time token pruning. At deployment we discard KV entries of the pruned heads to shrink the search space, then apply token-level context-reconstruction scoring only within the retained heads for fine-grained KV eviction (between prefill and decoding, following KVzip). This removes structurally redundant head-level computation while preserving the context-adaptive token selection that makes reconstruction attention query-agnostic — one compressed cache reused across many queries.

Matches full reconstruction across models & tasks

Accuracy vs KV cache ratio across tasks and models
Effect of KV cache ratio on accuracy for Qwen2.5-7B-1M, Qwen2.5-14B-1M and Qwen3-4B (SQuAD, GSM8K, Needle-in-a-Haystack, En.QA, and Average). HybridKV consistently matches or outperforms prior methods (KVzip, SnapKV, H2O, PyramidKV) at the same cache ratio, especially in low/mid-cache regimes. Unlike the head-only KVzip (Head) variant — which collapses at low cache ratios — HybridKV preserves accuracy by keeping token-level adaptivity within the salient heads.

Lower compression overhead

Efficiency analysis of KV cache compression
Efficiency analysis on Qwen2.5-7B-1M. By limiting expensive reconstruction scoring to the pre-selected salient heads, HybridKV cuts the additional compression-only overhead by up to 36% versus full context-reconstruction (KVzip) while retaining comparable accuracy. For a 124K-token context, full reconstruction adds 34s+ on top of a ~14s prefill (RTX PRO 6000, bf16).