Up to 2.1x faster decoding at 32k context in llama.cpp on an RTX 5090
Long AI conversations ran at about half speed on the newest NVIDIA graphics cards. I found why and fixed it in llama.cpp itself.
- 2.1x faster decoding
- at 32k context
- 1.7x faster decoding
- at 16k context
- About 10 lines changed
- in two functions
- 1728 of 1728 tests pass
- attention tests, with the fix
The problem
Attention ran at about half speed on an RTX 5090, a Blackwell GPU, when llama.cpp used a compressed q4_0 KV cache (the memory that holds each conversation's context). This happened in builds made with CUDA 12.8, the version in its Docker images.
It was not a recent bug. I compiled the kernel from llama.cpp as it stood at six dates going back a year, and every CUDA 12.8 build for Blackwell had it. It affects GeForce RTX 50 and RTX PRO Blackwell cards on llama.cpp's own Docker images and Ubuntu release builds. Older GPUs and newer CUDA versions were fine.
What I did
The CUDA 12.8 compiler kept a small value in slow local memory. I traced that and changed how the kernel reads it, in about 10 lines.
For engineers. On Blackwell, nvcc 12.8 gave the vector flash attention kernel a 128 to 336 byte stack frame when the V cache is q4_0 or q5_0. The trigger was a byte-pointer read of the dequantized word. The fix reads those bytes with shifts, so the value stays in registers and the stack frame is gone.
In review, a maintainer tested the change on older cards and found q5_0 running slower on an RTX 3090. I limited the new code path to Blackwell, where the problem is, and checked that older GPUs compile to identical code.
The result
At 32k tokens of context65.1 to 134.4 tok/s2.07x with the q4_0 cache
| Context (tokens) | Before | After | Change |
|---|---|---|---|
| 0 | 215.6 | 235.0 | 1.09x |
| 4k | 164.4 | 212.7 | 1.29x |
| 16k | 100.5 | 171.8 | 1.71x |
| 32k | 65.1 | 134.4 | 2.07x |
Measured with llama-bench, 128 generated tokens, llama.cpp master against the PR branch (PR #30077).
With the fix, the compressed q4_0 cache decodes at 97% of the speed of the uncompressed f16 cache at 16k context, against 57% before. The q8_0 cache is unchanged.
The fix is pull request #30077 (opens in a new tab), open and in review.
The full investigation, with the compiler output and the review, is in the write-up.