Reiner Pope – The math behind how LLMs are trained and served
LLM trainingLLM inferenceBatch sizeLatency optimizationCost analysisRoofline analysisMemory bandwidthCompute performanceKV CacheSparsityMixture of ExpertsExpert parallelismData center architectureScale up networkScale out networkPipeline parallelismMicrobatchingMemory capacityChinchilla scalingRL generationAPI pricingContext lengthCryptographic ciphersNeural network architectureReversible networks
Reiner Pope delivers a blackboard lecture on the mathematical and hardware principles behind training and serving large language models. He explains how batch size, sparsity, and various parallelism strategies (expert, pipeline) impact latency and cost, revealing the critical role of memory bandwidth and capacity in LLM performance. The discussion also touches on the economic implications of API pricing and the surprising parallels between neural network architectures and cryptographic ciphers.