Memory, GPU, CUDA, Inference
An interactive, memory-first curriculum for GPU programming and inference performance, from bytes and caches to kernels and tokens per second.
Why VisualGPU?
4 Topics, 36 Chapters, 1 Goal
NVIDIA H100 (Hopper) · SXM5
≈0.4 / 989.4 TFLOPS ≈ 0.04% of the chip — illustrative
Roadmap
Thirty-six chapters across four topics. Each chapter builds on the ones above it. Pick a topic to see its path, or All for the whole map. Drag to explore, click a chapter to jump in.
Curriculum
Four topics, one through-line: every chapter is a callback to the memory model you build first, from cache lines to coalescing to the KV cache.
Every performance phenomenon in this field is a memory phenomenon, so we learn memory first, deeply, on the CPU. Latency, bandwidth, caches, locality, and virtual memory become the vocabulary every later topic is a callback to: coalescing is cache lines, FlashAttention is tiling, paged KV is virtual memory.
The GPU is not “many cores”; it is the other answer to the memory wall: hide latency with more threads instead of more machinery. Understanding that trade, plus the roofline model, gives you the one lens that classifies every kernel and every model you will ever run as memory-bound or compute-bound.
Theory becomes muscle memory: you write one real kernel and optimize it step by step (coalescing, shared-memory tiling, tensor cores), watching its roofline dot climb after each change. The GEMM you build here is the exact computation transformers spend their lives inside, which is why it earns a whole arc.
The payoff: LLM serving is the memory problem par excellence. Prefill is compute-bound GEMM, decode is bandwidth-bound GEMV, and the KV cache decides what fits, so by the end you can predict tokens/sec for a model on a GPU from first principles, then benchmark it and reconcile the gap.
FAQ
Honest answers. I'm learning this alongside you and documenting everything as I go.