081 - How Does AI Inference Work?

Opublikowano: 2026-10-13

When you type a question into an AI assistant and watch the answer appear word by word, what's happening underneath is inference: the process of running a trained model to produce that answer. This episode follows one request through the full inference path, from tokenization to the KV cache to production serving. A prompt is first assembled with any system instructions, conversation history, and context, then converted by a tokenizer into token IDs. The model processes that token sequence through its transformer layers, including attention, producing logits, which are scores for...