Writing
What I find when something is slower than it should be, with the numbers and the dead ends.
- Half-speed attention on RTX 50 GPUs, caused by one pointer cast and CUDA 12.8On an RTX 5090, llama.cpp decoded at half speed with a 4-bit KV cache. The cause was how CUDA 12.8 compiles one byte read for Blackwell GPUs, and a ten-line change doubles decode speed at 32k context.
- FP8 was slower than BF16 on an H100. The cause was a divisibility check.Under load, FP8 Qwen3-8B served slower than BF16 on an H100 NVL with vLLM. The cause was one line of kernel selection, and fixing it gave 17 to 52% more output tokens per second on the CUTLASS FP8 path.
- Inference audit of Qwen3-8B on vLLM, one H100 NVLsample reportSample report in the format a client receives. It is not a client engagement and uses no client data. The deployment is my own test setup on a rented GPU, measured from 2026-10-03 to 2026-10-05. Upstream status was checked on 2026-10-11.