Back to selected work

Case study

Up to 2.1x faster decoding at 32k context in llama.cpp on an RTX 5090

Long AI conversations ran at about half speed on the newest NVIDIA graphics cards. I found why and fixed it in llama.cpp itself.

2.1x faster decoding
at 32k context
1.7x faster decoding
at 16k context
About 10 lines changed
in two functions
1728 of 1728 tests pass
attention tests, with the fix

The problem

Attention ran at about half speed on an RTX 5090, a Blackwell GPU, when llama.cpp used a compressed q4_0 KV cache (the memory that holds each conversation's context). This happened in builds made with CUDA 12.8, the version in its Docker images.

It was not a recent bug. I compiled the kernel from llama.cpp as it stood at six dates going back a year, and every CUDA 12.8 build for Blackwell had it. It affects GeForce RTX 50 and RTX PRO Blackwell cards on llama.cpp's own Docker images and Ubuntu release builds. Older GPUs and newer CUDA versions were fine.

What I did

The CUDA 12.8 compiler kept a small value in slow local memory. I traced that and changed how the kernel reads it, in about 10 lines.

beforeread valueuse valueslow memoryafterread valueuse valuekept in registers
Illustration of the fix

For engineers. On Blackwell, nvcc 12.8 gave the vector flash attention kernel a 128 to 336 byte stack frame when the V cache is q4_0 or q5_0. The trigger was a byte-pointer read of the dequantized word. The fix reads those bytes with shifts, so the value stays in registers and the stack frame is gone.

In review, a maintainer tested the change on older cards and found q5_0 running slower on an RTX 3090. I limited the new code path to Blackwell, where the problem is, and checked that older GPUs compile to identical code.

The result

Decode speed in tokens per second (higher is better) as the context grows. RTX 5090, CUDA 12.8.1, Qwen3-8B Q4_K_M, q4_0 cache

At 32k tokens of context65.1 to 134.4 tok/s2.07x with the q4_0 cache

Decode speed in tokens per second by context length, q4_0 cache. At 32k tokens of context it is 65.1 before the fix and 134.4 after.05010015020025004k16k32kafterbefore
Decode speed in tokens per second, q4_0 cache
Context (tokens)BeforeAfterChange
0215.6235.01.09x
4k164.4212.71.29x
16k100.5171.81.71x
32k65.1134.42.07x

Measured with llama-bench, 128 generated tokens, llama.cpp master against the PR branch (PR #30077).

before (llama.cpp master) after the fix

With the fix, the compressed q4_0 cache decodes at 97% of the speed of the uncompressed f16 cache at 16k context, against 57% before. The q8_0 cache is unchanged.

The fix is pull request #30077 (opens in a new tab), open and in review.

The full investigation, with the compiler output and the review, is in the write-up.

Ask about a first project