Exploring KV Cache Quantization in Multimodal Large Language Model Inference

Who Should Read

Engineers optimizing multimodal LLM inference for production deployments who need to reduce latency and memory footprint without significant quality loss.

Core Mechanics

• The KV (Key-Value) Cache for multimodal inputs is significantly larger than for text-only inputs because image tokens dominate
• Standard KV Cache quantization (INT8/INT4) degrades multimodal quality more than text-only quality — image features are more sensitive to quantization noise
• The paper proposes modality-aware KV quantization: higher precision for image token KV entries, lower precision for text token KV entries
• This mixed-precision approach achieves 1.7x improvement in time-to-first-token and 4.3x improvement in generation throughput
• Quality degradation is minimal: < 1% on VQA benchmarks, < 2% on image captioning tasks
• The memory savings from KV cache quantization allow processing 3x longer multimodal contexts within the same memory budget

Evidence

• Time-to-first-token: baseline 2.4s → quantized 1.4s (1.7x speedup)
• Output throughput: baseline 42 tokens/s → quantized 181 tokens/s (4.3x speedup)
• VQA accuracy drop: INT8 modality-aware quantization shows 0.8% accuracy loss vs. 3.2% for uniform INT8 quantization

How to Apply

• For vLLM or TensorRT-LLM deployments: implement modality-aware KV cache quantization by identifying which KV cache entries correspond to image tokens and applying INT8 to text entries, INT4 or FP8 to image entries — or vice versa based on your quality requirements.
• The quantization benefit is largest when image token counts are high — if you're processing many images or high-resolution inputs, prioritize this optimization.
• Profile your specific model and hardware combination: the optimal precision split varies — start with INT8/INT4 text/image and benchmark quality vs. throughput tradeoffs.

Code Example

# Conceptual application example (PyTorch pseudo-code)
import torch

def mixed_precision_kv_cache(keys, values, text_token_mask, quant_bits=4):
    """
    text_token_mask: True positions are text tokens (top 10%)
    Image tokens are quantized to lower bits
    """
    # Text tokens: maintain high precision
    keys_text = keys[text_token_mask]      # Keep as FP16
    values_text = values[text_token_mask]

    # Image tokens: INT4 quantization
    keys_img = keys[~text_token_mask]
    values_img = values[~text_token_mask]

    scale_k = keys_img.abs().max() / (2 ** (quant_bits - 1) - 1)
    keys_img_q = (keys_img / scale_k).round().to(torch.int8)

    scale_v = values_img.abs().max() / (2 ** (quant_bits - 1) - 1)
    values_img_q = (values_img / scale_v).round().to(torch.int8)

    return {
        "keys_text": keys_text,
        "values_text": values_text,
        "keys_img_quantized": keys_img_q,
        "keys_img_scale": scale_k,
        "values_img_quantized": values_img_q,
        "values_img_scale": scale_v,
    }

Terminology