Try adjusting your filters or search query
This tool never shows a number without saying how confident it is. A figure with no symbol next to it was computed from the model's own published config; a figure with a symbol was computed from something less direct, and the symbol tells you exactly what.
Every VRAM and KV-cache figure here is arithmetic — parameter count × bytes per parameter, attention geometry × tokens — not a reading from a real vLLM process. Use them to shortlist hardware, then confirm by actually starting the server.
The model's own HuggingFace repo is gated, so its attention geometry (layers, KV heads, head dim, sliding window) was read from an ungated repo of the same architecture. Almost always right — but it is a stand-in, not the model's own file.
No config.json was reachable anywhere — gated repo, no mirror. The
geometry was entered by hand from the model card or paper. Treat the KV number as
a rough guide and verify before you size hardware around it.
The weakest number on the site. With no attention geometry at all, the KV cache is approximated straight from parameter count and context length. It ignores GQA and MLA (which shrink the cache a lot) and over-estimates MoE models, so it usually reads too high. Don't rule a model out on this figure alone.
vLLM sizes the KV cache as budget − weights − the activation and CUDA-graph peak it
measures at startup by profiling a dummy forward pass. That peak scales with
--max-num-batched-tokens and the model's hidden size, and there is no
sourced way to estimate it here, so it is not modelled. A fit with only a few GB to
spare may not survive it — such fits are shown as an amber
✓ (tight); lower
Mem util to see a tighter server.
The quantized weights load and run on this GPU, but the GPU has no tensor cores for that format — vLLM dequantizes on the fly (e.g. INT4/AWQ via Marlin). You get the memory saving, not a speedup, and throughput can be worse than a natively-supported format. A plain ✓ means native hardware support.
✓ = model weights (plus the KV cache, if you switched the KV estimate on) sit inside the budget: physical VRAM × memory utilization. ✗ = too large, or the quantization format isn't supported on that GPU at all. Hover any verdict for the arithmetic behind it.
Weights (plus KV, if enabled) fit the budget, but with under
8 GB per GPU to spare. That figure is a heuristic warning
band, not a measurement, and nothing is subtracted from the budget: vLLM
measures the real activation and CUDA-graph peak at startup by profiling a dummy forward
pass, and real logs put that peak anywhere from about 0.1 GiB (a 0.5B model) to
18 GiB per GPU (Gemma-3-27B at a 128K max_model_len), so a thin margin
may not survive it. If you see an amber ✓, lower --max-num-batched-tokens
or --max-model-len, pick a smaller quantization, or add a GPU.
Full explanation with figures: How vLLM uses GPU memory (startup profiling, KV pool sizing, scheduler, every flag).
A card's bar splits the selected GPUs' usable memory into weights, KV cache and free space — red when it overflows. It is drawn from the same fit result that decides the ✓/✗, so the bar and the verdict can never disagree.
Some or all of the model's layers cap their KV cache at a fixed window instead of growing with the full context, so its KV stays far smaller than a full-attention model of the same size. A surprisingly small KV number on an SWA model is the design, not a bug.
Diffusion-style and other single-pass models have no autoregressive KV cache at all — context length costs them no extra VRAM. Their KV row reads zero on purpose.
Want the hardware side — which quantization formats run natively on which GPU, and how the memory budget is derived? Open .