Speculative Decoding in vLLM on AMD GPUs
vLLM published a benchmark-heavy guide to speculative decoding on AMD Instinct MI300X/MI355X GPUs (ROCm), comparing five drafting approaches — native and separate MTP modules plus dedicated speculator networks (EAGLE-3, DFlash, DSpark) — against baseline autoregressive decoding across GSM8K, MATH500, HumanEval, and MBPP. Speedups varied sharply by model and method: up to 2.87x on Gemma 4 26B-A4B and 2.68x on Kimi-K2.5 with DFlash on MATH500, while Qwen models ranged from a modest 1.08x to 2.20x depending on drafter and proposal length. It's not a new vLLM release — speculative decoding already existed — but it's a rare task-grounded, tunable-by-tunable look at squeezing real throughput out of AMD hardware, an area with much thinner public benchmarking than NVIDIA gets.
Sources & depth
- vllm.aiSpeculative Decoding in vLLM on AMD GPUs2026-09-07 14:56 IST