KV Cache Calculator

This tool allows you to calculate the memory footprint of the KV cache for large language models. It supports various model architectures including grouped-query attention (GQA), multi-query attention (MQA), latent attention (MLA), sliding-window attention, and hybrid models. You can specify parameters such as the model configuration (read from Hugging Face), serving engine (vLLM, SGLang, TensorRT-LLM), cache precision (BF16, FP8, FP4), context tokens per request, and concurrent requests. The calculator also considers tensor parallelism and the prefill-to-decode handoff rate.

• Model Configuration: Input model names or repository IDs to fetch configuration from Hugging Face.
• Serving Engine Options: Choose between vLLM, SGLang, and TensorRT-LLM, each with different memory utilization rules.
• Cache Precision: Select from BF16, FP8, or FP4 to determine memory usage per element.
• Request Parameters: Define the number of tokens per request and the number of concurrent requests.
• GPU and Parallelism: Specify GPU memory, memory share, and tensor-parallel degrees.
• Handoff Rates: Input custom or select standard rates for inter-GPU communication (NVLink, PCIe, Ethernet).

The tool provides detailed breakdowns of cache size per token, per request, and per GPU, helping you understand and optimize memory usage for efficient LLM deployment.