I thought I understood how GPU loading works until I started writing this guide. 🤔
I was constantly confusing three different concepts:
• The amount of work we are launching.
• The amount of work that is actually placed on the GPU.
• The amount of work that is ready to be executed right now.
But these are completely different things. ⚠️
Consider a CUDA kernel with 240 blocks, each containing 256 threads. This results in 61,440 logical threads, or 1920 warps. However, the number of launched threads doesn't tell you how many warps can actually be present on the GPU simultaneously. 🧮
Blocks are placed on streaming multiprocessors (SMs) based on the available registers, shared memory, and limitations on the number of threads, warps, and blocks.
Even after being placed on an SM, a warp can be waiting for a memory operation to complete, a dependency between arithmetic operations to be resolved, or for synchronization. The scheduler can only select warps that are actually ready to issue instructions for execution. ⏳
This changed my approach to analyzing performance metrics. 📊
Occupancy is a measure of the ratio of the number of active warps on an SM to the maximum number of warps that the SM can support.
GPU utilization measures something different. In NVML, this metric reflects the percentage of the measurement interval during which at least one CUDA kernel was running.
Therefore, the GPU can show 100% utilization even if the Tensor Cores are not being used to their full potential, the HBM bandwidth is not saturated, and performance is limited by one of the system components. At the same time, a significant portion of the GPU's resources may remain underutilized. 📉
Similar misconceptions arise with memory. Registers, shared memory, L1, L2, and HBM are not simply levels of memory that differ in speed. Each has its own areas of accessibility, capacity, management characteristics, and access methods. 💾
If a model fits in HBM, it only means that there is enough video memory for it. But it tells you very little about how many bytes need to be transferred, how efficiently memory accesses are combined, how often data is reused, and whether memory is actually limiting the performance of the CUDA kernel.
A good example is FlashAttention. This algorithm calculates the exact attention mechanism, but uses tiling and repeated computations to reduce the amount of data transferred between HBM and the GPU's internal memory. ⚡️
Tensor Cores turned out to be another technology that I previously understood too simplistically.
These are specialized computational units for matrix multiplication operations with accumulation, which play a huge role in AI tasks. However, they do not "execute the model" as a whole. 🧠
Reduction operations, indexing, element-wise calculations, synchronization, and data movement still involve other components of the GPU. In addition, launching CUDA kernels also introduces overhead.
Over time, I stopped asking myself: "Why isn't the GPU 100% utilized?" Instead, I started asking a more accurate question: "What is currently limiting the speed of performing useful work?" 🤔
Ultimately, all of this turned into a 42-page guide. 📄
It covers streaming multiprocessors (SMs), warps, and schedulers, the memory hierarchy and latency hiding, Tensor Cores, the distribution of GEMM computations across GPU hardware resources, numerical precision, and performance metrics that are very easy to misinterpret.
https://drive.google.com/file/d/17tyvxr__8vZ6o995GNq6I5PC5JOiv0ql/view
#GPU #CUDA #AI #DeepLearning #PerformanceOptimization #TechGuide
✨ Join Best TG Channels https://t.iss.one/addlist/0f6vfFbEMdAwODBk
⭐️ Join Our WhatsApp Channel https://whatsapp.com/channel/0029VaC7Weq29753hpcggW2A
I was constantly confusing three different concepts:
• The amount of work we are launching.
• The amount of work that is actually placed on the GPU.
• The amount of work that is ready to be executed right now.
But these are completely different things. ⚠️
Consider a CUDA kernel with 240 blocks, each containing 256 threads. This results in 61,440 logical threads, or 1920 warps. However, the number of launched threads doesn't tell you how many warps can actually be present on the GPU simultaneously. 🧮
Blocks are placed on streaming multiprocessors (SMs) based on the available registers, shared memory, and limitations on the number of threads, warps, and blocks.
Even after being placed on an SM, a warp can be waiting for a memory operation to complete, a dependency between arithmetic operations to be resolved, or for synchronization. The scheduler can only select warps that are actually ready to issue instructions for execution. ⏳
This changed my approach to analyzing performance metrics. 📊
Occupancy is a measure of the ratio of the number of active warps on an SM to the maximum number of warps that the SM can support.
GPU utilization measures something different. In NVML, this metric reflects the percentage of the measurement interval during which at least one CUDA kernel was running.
Therefore, the GPU can show 100% utilization even if the Tensor Cores are not being used to their full potential, the HBM bandwidth is not saturated, and performance is limited by one of the system components. At the same time, a significant portion of the GPU's resources may remain underutilized. 📉
Similar misconceptions arise with memory. Registers, shared memory, L1, L2, and HBM are not simply levels of memory that differ in speed. Each has its own areas of accessibility, capacity, management characteristics, and access methods. 💾
If a model fits in HBM, it only means that there is enough video memory for it. But it tells you very little about how many bytes need to be transferred, how efficiently memory accesses are combined, how often data is reused, and whether memory is actually limiting the performance of the CUDA kernel.
A good example is FlashAttention. This algorithm calculates the exact attention mechanism, but uses tiling and repeated computations to reduce the amount of data transferred between HBM and the GPU's internal memory. ⚡️
Tensor Cores turned out to be another technology that I previously understood too simplistically.
These are specialized computational units for matrix multiplication operations with accumulation, which play a huge role in AI tasks. However, they do not "execute the model" as a whole. 🧠
Reduction operations, indexing, element-wise calculations, synchronization, and data movement still involve other components of the GPU. In addition, launching CUDA kernels also introduces overhead.
Over time, I stopped asking myself: "Why isn't the GPU 100% utilized?" Instead, I started asking a more accurate question: "What is currently limiting the speed of performing useful work?" 🤔
Ultimately, all of this turned into a 42-page guide. 📄
It covers streaming multiprocessors (SMs), warps, and schedulers, the memory hierarchy and latency hiding, Tensor Cores, the distribution of GEMM computations across GPU hardware resources, numerical precision, and performance metrics that are very easy to misinterpret.
https://drive.google.com/file/d/17tyvxr__8vZ6o995GNq6I5PC5JOiv0ql/view
#GPU #CUDA #AI #DeepLearning #PerformanceOptimization #TechGuide
✨ Join Best TG Channels https://t.iss.one/addlist/0f6vfFbEMdAwODBk
⭐️ Join Our WhatsApp Channel https://whatsapp.com/channel/0029VaC7Weq29753hpcggW2A
👍2❤1