KV cache is the real inference bottleneck (Not GPUs)

Podcast: Pop Goes the Stack · F5

Opublikowano: 2026-05-05

GPUs get all the attention, but in inference, the real bottleneck is often memory, specifically the KV cache. In this episode of Pop Goes the Stack, Lori MacVittie sits down with Tim Michels to explain why inference stopped being stateless the moment long contexts, multi-turn conversations, and never-ending agents became normal. That state has to live somewhere, and too often it’s living in the most expensive place in the stack. Tim breaks down what KV cache actually is by separating inference into its two phases: prefill, where prompts are tokenized and transformed into th...