Ollama v0.40.0 makes MLX the default runtime on Apple Silicon
Ollama's v0.40.0 release candidate says models now run on Apple's MLX framework by default on Apple Silicon Macs, with MLX support extended to more model families, including Qwen and Gemma variants. The same release series also tightens how MLX tokenizers match what model publishers ship (pre-tokenizer order, split behavior, added-token handling), a correctness fix that matters when outputs are compared across runtimes. It is still labelled a release candidate (rc1 through rc3 appeared within a day), so the default may shift before a final tag.
Sources & depth
Helion kernels beat CUTLASS and DeepGEMM in vLLM on Hopper GPUs
PyTorch and Red Hat describe a vLLM linear backend built on Helion, a PyTorch kernel language that autotunes per input shape. On an H100, tuned FP8 and INT8 matrix-multiply kernels were 1.11 to 1.18 times faster (geometric mean) than vLLM's default CUTLASS, FlashInfer and DeepGEMM kernels, and end-to-end serving throughput rose by more than 10% on some workloads. The gains cover small batches only (up to 32 tokens, the decoding regime), and tuning takes hours per model, so the backend ships in a Red Hat vLLM fork rather than upstream vLLM; the authors propose having users run the tuner themselves.
Sources & depth
Strata: a community engine for running a 125B Qwen 3.8 Flash Next on gaming GPUs
An open-source project called Strata claims to run Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on consumer NVIDIA and AMD cards with 12 GB or more of VRAM by spreading experts across GPU, RAM and SSD. Its README tables list 53 to 94 tokens per second on an RTX 5070 at heavily quantized levels (2 to 3 bits); the 100 tokens per second on an RTX 4090 in the Hacker News title does not appear there. These are the author's own unreplicated numbers on aggressively compressed weights, so treat them as a lead rather than a result.
Sources & depth