083 - Why LLMs Are Slow: Inside GPU Inference

Opublikowano: 2026-10-15

When an AI model answers slowly, the problem usually isn't its intelligence — it's how the serving system moves data through a GPU. This episode goes one level below the basic inference loop covered in an earlier episode, into GPU memory, the key-value cache, attention kernels, and the scheduling techniques used by systems like vLLM and NVIDIA TensorRT-LLM. It starts by separating prefill, which processes prompt tokens in parallel and benefits from raw compute throughput, from decode, which is autoregressive and becomes memory-bound because each step needs the model weights and the gr...